{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML with Tabular data - Training Models and Ensembling \n",
    "\n",
    "We next focus on the training of individual models and how to combine their predictions into an ensemble that is more accurate than any of the individual models. \n",
    "\n",
    "**Motivating Example:** Here we assume you have some basic familiarity with the concepts of [Decision Trees](https://bradleyboehmke.github.io/HOML/DT.html) and [Random Forests](https://bradleyboehmke.github.io/HOML/random-forest.html), but this knowledge is not required for the rest of the tutorial. \n",
    "Consider the dataset shown below in panel **(A)** with 2 features (one per axis), involving a regression task with the color intensity of each datapoint indicating values of the target-variable.  If we train a decision tree model on this data, the tree splits might be selected as in **(B)**, with the resulting  regression function learned by the decision tree shown in **(C)**.\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "     <td style=\"text-align:center\"> <img src=\"files/images/EnsembleExampleData.png\" width=\"200\" height=\"200\"/> </td>\n",
    "    <td style=\"text-align:center\"> <img src=\"files/images/EnsembleExampleDTwData.png\" width=\"200\" height=\"200\"/>  </td>\n",
    "    <td style=\"text-align:center\"> <img src=\"files/images/EnsembleExampleDT.png\" width=\"200\" height=\"200\"/> </td>\n",
    "      <td style=\"text-align:center\"> <img src=\"files/images/EnsembleExampleDTensemble.png\" width=\"200\" height=\"200\"/>  </td>\n",
    "    <td style=\"text-align:center\"> <img src=\"files/images/EnsembleExampleRF.png\" width=\"200\" height=\"200\"/> </td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td style=\"text-align:center\"> (A) Dataset </td> \n",
    "      <td style=\"text-align:center\"> (B) Decision Tree Splits </td> \n",
    "      <td style=\"text-align:center\"> (C) Decision Tree Predictions </td> \n",
    "      <td style=\"text-align:center\"> (D) 4 Decision Tree Models </td> \n",
    "      <td style=\"text-align:center\"> (E) Ensemble Predictions </td> \n",
    "  </tr>\n",
    "</table>\n",
    "\n",
    "We can train multiple decision tree models as shown in **(D)**, obtaining different predictors in each training run if some randomness is introduced (eg. via: subsampling of the data used to fit each tree, or random-selection of feature in each split). If we simply average the predictions of each decision tree, the resulting ensemble-model produces predictions shown in **(E)**. Note that these ensemble predictions appear much better suited for the data from **(A)**, and such ensembling is the strategy Random Forests use to produce much higher accuracy than individual decision trees.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Desirable Ensemble Characteristics:** Let $f_1, f_2, \\dots, f_{9}$ denote 9 different trained predictors, where $f_j(x)$ is the $j$th model's prediction for a datapoint with feature-values $x$ (predicted class-probabilities in classification).  We can construct an ensemble-predictor $f_*(x) = \\frac{1}{9} \\sum_{j=1}^{9} f_j(x)$ by averaging the predictions of the individual models as graphically depicted in **(E)** above. \n",
    "\n",
    "Suppose we are doing binary classification and each individual predictor $f_j$ is extremely-confident outputting either 0 or 1 for its estimated probability that a datapoint belongs to the positive class. Assume each $f_j$ \n",
    "has error-rate = 0.3, that is, there is a 30% chance this predictor makes the wrong prediction given a new datapoint sampled from the underlying data-generating distribution.\n",
    "\n",
    "If the predictions of $f_1, f_2, \\dots, f_{9}$ are completely identical for all $x$ (i.e. these predictors make highly correlated errors), then the error-rate of $f_*$ will also obviously be 0.3 and nothing has been gained from ensembling.  At the opposite extreme, suppose these predictors make statistically independent errors. In this case, $f_*(x)$ only makes an error if at least 5 of the predictors are wrong, which happens with probability $\\displaystyle = \\sum_{k=5}^{9} {9 \\choose k} 0.3^k (1 - 0.3)^{9-k} \\le 0.1$, so the error-rate of the ensemble is far lower than the error-rate of the individual predictors.\n",
    "\n",
    "In general, the accuracy of our ensemble will improve the more individual predictors we combine and the less-correlated their errors are. Two ways to reduce this correlation, i.e. increase the *diversity* of the ensemble, are: \n",
    "\n",
    "1) Train completely different types of models that tend to learn different types of functions.\n",
    "\n",
    "2) Train different models using different subsets of the data. \n",
    "\n",
    "In practice, our individual predictors $f_1, f_2, \\dots, f_{9}$ will have different error-rates, in which case we prefer to construct a *weighted* ensemble: $f_*(x) = \\sum_{j=1}^{9} w_j f_j(x)$, where the weights $w_j$ sum to 1 and are chosen to favor the more-accurate predictors. To learn more about different ensembling strategies, see:  ([Dietterich, 2000](https://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf); [Rocca, 2019](https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205)).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training different types of models:\n",
    "\n",
    "Let's revisit what happens in AutoGluon training by default, this time specifying `'balanced_accuracy'` as the evaluation-metric AutoGluon should optimize for, and increasing the `verbosity` to print more training details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv | Columns = 47 / 47 | Rows = 61059 -> 61059\n",
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_200529/\n",
      "Beginning AutoGluon training ...\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_200529/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 600 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Total time taken in fit_transform: 0.31229400634765625\n",
      "\tData preprocessing and feature engineering runtime = 0.34s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: balanced_accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: balanced_accuracy\n",
      "Saving AutogluonModels/ag-20200801_200529/learner.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/utils/data/X_train.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/utils/data/y_train.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/utils/data/X_val.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/utils/data/y_val.pkl\n",
      "Fitting model: RandomForestClassifierGini ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/RandomForestClassifierGini/model.pkl\n",
      "\t0.4311\t = Validation balanced_accuracy score\n",
      "\t0.7s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: RandomForestClassifierEntr ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/RandomForestClassifierEntr/model.pkl\n",
      "\t0.4422\t = Validation balanced_accuracy score\n",
      "\t0.65s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierGini ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierGini/model.pkl\n",
      "\t0.4411\t = Validation balanced_accuracy score\n",
      "\t0.55s\t = Training runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierEntr ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierEntr/model.pkl\n",
      "\t0.46\t = Validation balanced_accuracy score\n",
      "\t0.54s\t = Training runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierUnif ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierUnif/model.pkl\n",
      "\t0.3878\t = Validation balanced_accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierDist ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierDist/model.pkl\n",
      "\t0.41\t = Validation balanced_accuracy score\n",
      "\t0.01s\t = Training runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: LightGBMClassifier ...\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttrain_set's multi_logloss: 0.89897\ttrain_set's balanced_accuracy: 0.333333\tvalid_set's multi_logloss: 0.907116\tvalid_set's balanced_accuracy: 0.333333\n",
      "[2]\ttrain_set's multi_logloss: 0.880115\ttrain_set's balanced_accuracy: 0.360721\tvalid_set's multi_logloss: 0.899548\tvalid_set's balanced_accuracy: 0.354444\n",
      "[3]\ttrain_set's multi_logloss: 0.861484\ttrain_set's balanced_accuracy: 0.406287\tvalid_set's multi_logloss: 0.894264\tvalid_set's balanced_accuracy: 0.384444\n",
      "[4]\ttrain_set's multi_logloss: 0.843264\ttrain_set's balanced_accuracy: 0.455986\tvalid_set's multi_logloss: 0.895379\tvalid_set's balanced_accuracy: 0.367778\n",
      "[5]\ttrain_set's multi_logloss: 0.825671\ttrain_set's balanced_accuracy: 0.478297\tvalid_set's multi_logloss: 0.890938\tvalid_set's balanced_accuracy: 0.388889\n",
      "[6]\ttrain_set's multi_logloss: 0.810194\ttrain_set's balanced_accuracy: 0.496161\tvalid_set's multi_logloss: 0.890804\tvalid_set's balanced_accuracy: 0.377778\n",
      "[7]\ttrain_set's multi_logloss: 0.793599\ttrain_set's balanced_accuracy: 0.509698\tvalid_set's multi_logloss: 0.892886\tvalid_set's balanced_accuracy: 0.397778\n",
      "[8]\ttrain_set's multi_logloss: 0.779873\ttrain_set's balanced_accuracy: 0.521228\tvalid_set's multi_logloss: 0.894467\tvalid_set's balanced_accuracy: 0.391111\n",
      "[9]\ttrain_set's multi_logloss: 0.764224\ttrain_set's balanced_accuracy: 0.521542\tvalid_set's multi_logloss: 0.891809\tvalid_set's balanced_accuracy: 0.406667\n",
      "[10]\ttrain_set's multi_logloss: 0.750265\ttrain_set's balanced_accuracy: 0.524178\tvalid_set's multi_logloss: 0.888637\tvalid_set's balanced_accuracy: 0.413333\n",
      "[11]\ttrain_set's multi_logloss: 0.737528\ttrain_set's balanced_accuracy: 0.524807\tvalid_set's multi_logloss: 0.882844\tvalid_set's balanced_accuracy: 0.4\n",
      "[12]\ttrain_set's multi_logloss: 0.724306\ttrain_set's balanced_accuracy: 0.538344\tvalid_set's multi_logloss: 0.878551\tvalid_set's balanced_accuracy: 0.4\n",
      "[13]\ttrain_set's multi_logloss: 0.713158\ttrain_set's balanced_accuracy: 0.538973\tvalid_set's multi_logloss: 0.875068\tvalid_set's balanced_accuracy: 0.4\n",
      "[14]\ttrain_set's multi_logloss: 0.702141\ttrain_set's balanced_accuracy: 0.544049\tvalid_set's multi_logloss: 0.870817\tvalid_set's balanced_accuracy: 0.396667\n",
      "[15]\ttrain_set's multi_logloss: 0.687553\ttrain_set's balanced_accuracy: 0.556956\tvalid_set's multi_logloss: 0.8704\tvalid_set's balanced_accuracy: 0.421111\n",
      "[16]\ttrain_set's multi_logloss: 0.676857\ttrain_set's balanced_accuracy: 0.557271\tvalid_set's multi_logloss: 0.865934\tvalid_set's balanced_accuracy: 0.407778\n",
      "[17]\ttrain_set's multi_logloss: 0.666426\ttrain_set's balanced_accuracy: 0.557586\tvalid_set's multi_logloss: 0.862024\tvalid_set's balanced_accuracy: 0.418889\n",
      "[18]\ttrain_set's multi_logloss: 0.655887\ttrain_set's balanced_accuracy: 0.56034\tvalid_set's multi_logloss: 0.860642\tvalid_set's balanced_accuracy: 0.418889\n",
      "[19]\ttrain_set's multi_logloss: 0.644565\ttrain_set's balanced_accuracy: 0.565102\tvalid_set's multi_logloss: 0.861389\tvalid_set's balanced_accuracy: 0.405556\n",
      "[20]\ttrain_set's multi_logloss: 0.634854\ttrain_set's balanced_accuracy: 0.570178\tvalid_set's multi_logloss: 0.861099\tvalid_set's balanced_accuracy: 0.401111\n",
      "[21]\ttrain_set's multi_logloss: 0.626362\ttrain_set's balanced_accuracy: 0.572185\tvalid_set's multi_logloss: 0.858535\tvalid_set's balanced_accuracy: 0.431111\n",
      "[22]\ttrain_set's multi_logloss: 0.614655\ttrain_set's balanced_accuracy: 0.609167\tvalid_set's multi_logloss: 0.85999\tvalid_set's balanced_accuracy: 0.4\n",
      "[23]\ttrain_set's multi_logloss: 0.605034\ttrain_set's balanced_accuracy: 0.614543\tvalid_set's multi_logloss: 0.858048\tvalid_set's balanced_accuracy: 0.42\n",
      "[24]\ttrain_set's multi_logloss: 0.593704\ttrain_set's balanced_accuracy: 0.640326\tvalid_set's multi_logloss: 0.862752\tvalid_set's balanced_accuracy: 0.413333\n",
      "[25]\ttrain_set's multi_logloss: 0.58312\ttrain_set's balanced_accuracy: 0.65184\tvalid_set's multi_logloss: 0.860432\tvalid_set's balanced_accuracy: 0.413333\n",
      "[26]\ttrain_set's multi_logloss: 0.573092\ttrain_set's balanced_accuracy: 0.676246\tvalid_set's multi_logloss: 0.861967\tvalid_set's balanced_accuracy: 0.413333\n",
      "[27]\ttrain_set's multi_logloss: 0.564467\ttrain_set's balanced_accuracy: 0.682998\tvalid_set's multi_logloss: 0.863907\tvalid_set's balanced_accuracy: 0.406667\n",
      "[28]\ttrain_set's multi_logloss: 0.554636\ttrain_set's balanced_accuracy: 0.690829\tvalid_set's multi_logloss: 0.86499\tvalid_set's balanced_accuracy: 0.413333\n",
      "[29]\ttrain_set's multi_logloss: 0.546333\ttrain_set's balanced_accuracy: 0.710159\tvalid_set's multi_logloss: 0.863624\tvalid_set's balanced_accuracy: 0.42\n",
      "[30]\ttrain_set's multi_logloss: 0.537562\ttrain_set's balanced_accuracy: 0.719667\tvalid_set's multi_logloss: 0.864469\tvalid_set's balanced_accuracy: 0.42\n",
      "[31]\ttrain_set's multi_logloss: 0.528559\ttrain_set's balanced_accuracy: 0.749432\tvalid_set's multi_logloss: 0.865666\tvalid_set's balanced_accuracy: 0.426667\n",
      "[32]\ttrain_set's multi_logloss: 0.520981\ttrain_set's balanced_accuracy: 0.756948\tvalid_set's multi_logloss: 0.867282\tvalid_set's balanced_accuracy: 0.42\n",
      "[33]\ttrain_set's multi_logloss: 0.512694\ttrain_set's balanced_accuracy: 0.772894\tvalid_set's multi_logloss: 0.869256\tvalid_set's balanced_accuracy: 0.407778\n",
      "[34]\ttrain_set's multi_logloss: 0.504527\ttrain_set's balanced_accuracy: 0.799291\tvalid_set's multi_logloss: 0.866566\tvalid_set's balanced_accuracy: 0.426667\n",
      "[35]\ttrain_set's multi_logloss: 0.497026\ttrain_set's balanced_accuracy: 0.812182\tvalid_set's multi_logloss: 0.868587\tvalid_set's balanced_accuracy: 0.453333\n",
      "[36]\ttrain_set's multi_logloss: 0.489485\ttrain_set's balanced_accuracy: 0.807421\tvalid_set's multi_logloss: 0.867783\tvalid_set's balanced_accuracy: 0.453333\n",
      "[37]\ttrain_set's multi_logloss: 0.482198\ttrain_set's balanced_accuracy: 0.815566\tvalid_set's multi_logloss: 0.868212\tvalid_set's balanced_accuracy: 0.46\n",
      "[38]\ttrain_set's multi_logloss: 0.474776\ttrain_set's balanced_accuracy: 0.820328\tvalid_set's multi_logloss: 0.867821\tvalid_set's balanced_accuracy: 0.466667\n",
      "[39]\ttrain_set's multi_logloss: 0.467965\ttrain_set's balanced_accuracy: 0.843341\tvalid_set's multi_logloss: 0.864792\tvalid_set's balanced_accuracy: 0.473333\n",
      "[40]\ttrain_set's multi_logloss: 0.460089\ttrain_set's balanced_accuracy: 0.845033\tvalid_set's multi_logloss: 0.865046\tvalid_set's balanced_accuracy: 0.473333\n",
      "[41]\ttrain_set's multi_logloss: 0.452808\ttrain_set's balanced_accuracy: 0.864677\tvalid_set's multi_logloss: 0.864053\tvalid_set's balanced_accuracy: 0.466667\n",
      "[42]\ttrain_set's multi_logloss: 0.445846\ttrain_set's balanced_accuracy: 0.874185\tvalid_set's multi_logloss: 0.862205\tvalid_set's balanced_accuracy: 0.461111\n",
      "[43]\ttrain_set's multi_logloss: 0.43919\ttrain_set's balanced_accuracy: 0.892137\tvalid_set's multi_logloss: 0.863997\tvalid_set's balanced_accuracy: 0.455556\n",
      "[44]\ttrain_set's multi_logloss: 0.432206\ttrain_set's balanced_accuracy: 0.892137\tvalid_set's multi_logloss: 0.865873\tvalid_set's balanced_accuracy: 0.455556\n",
      "[45]\ttrain_set's multi_logloss: 0.425576\ttrain_set's balanced_accuracy: 0.916527\tvalid_set's multi_logloss: 0.863213\tvalid_set's balanced_accuracy: 0.455556\n",
      "[46]\ttrain_set's multi_logloss: 0.419668\ttrain_set's balanced_accuracy: 0.926349\tvalid_set's multi_logloss: 0.866934\tvalid_set's balanced_accuracy: 0.467778\n",
      "[47]\ttrain_set's multi_logloss: 0.413134\ttrain_set's balanced_accuracy: 0.918219\tvalid_set's multi_logloss: 0.86859\tvalid_set's balanced_accuracy: 0.467778\n",
      "[48]\ttrain_set's multi_logloss: 0.406649\ttrain_set's balanced_accuracy: 0.928041\tvalid_set's multi_logloss: 0.868282\tvalid_set's balanced_accuracy: 0.467778\n",
      "[49]\ttrain_set's multi_logloss: 0.400646\ttrain_set's balanced_accuracy: 0.934479\tvalid_set's multi_logloss: 0.866508\tvalid_set's balanced_accuracy: 0.474444\n",
      "[50]\ttrain_set's multi_logloss: 0.394753\ttrain_set's balanced_accuracy: 0.935857\tvalid_set's multi_logloss: 0.868852\tvalid_set's balanced_accuracy: 0.463333\n",
      "[51]\ttrain_set's multi_logloss: 0.389809\ttrain_set's balanced_accuracy: 0.945679\tvalid_set's multi_logloss: 0.869111\tvalid_set's balanced_accuracy: 0.462222\n",
      "[52]\ttrain_set's multi_logloss: 0.383533\ttrain_set's balanced_accuracy: 0.945993\tvalid_set's multi_logloss: 0.8675\tvalid_set's balanced_accuracy: 0.502222\n",
      "[53]\ttrain_set's multi_logloss: 0.378531\ttrain_set's balanced_accuracy: 0.945993\tvalid_set's multi_logloss: 0.868998\tvalid_set's balanced_accuracy: 0.502222\n",
      "[54]\ttrain_set's multi_logloss: 0.373501\ttrain_set's balanced_accuracy: 0.950126\tvalid_set's multi_logloss: 0.867429\tvalid_set's balanced_accuracy: 0.474444\n",
      "[55]\ttrain_set's multi_logloss: 0.368324\ttrain_set's balanced_accuracy: 0.959948\tvalid_set's multi_logloss: 0.868855\tvalid_set's balanced_accuracy: 0.502222\n",
      "[56]\ttrain_set's multi_logloss: 0.363145\ttrain_set's balanced_accuracy: 0.968393\tvalid_set's multi_logloss: 0.864494\tvalid_set's balanced_accuracy: 0.507778\n",
      "[57]\ttrain_set's multi_logloss: 0.358397\ttrain_set's balanced_accuracy: 0.96977\tvalid_set's multi_logloss: 0.864541\tvalid_set's balanced_accuracy: 0.502222\n",
      "[58]\ttrain_set's multi_logloss: 0.353482\ttrain_set's balanced_accuracy: 0.96977\tvalid_set's multi_logloss: 0.864229\tvalid_set's balanced_accuracy: 0.502222\n",
      "[59]\ttrain_set's multi_logloss: 0.348844\ttrain_set's balanced_accuracy: 0.96977\tvalid_set's multi_logloss: 0.865654\tvalid_set's balanced_accuracy: 0.496667\n",
      "[60]\ttrain_set's multi_logloss: 0.344215\ttrain_set's balanced_accuracy: 0.979592\tvalid_set's multi_logloss: 0.866386\tvalid_set's balanced_accuracy: 0.502222\n",
      "[61]\ttrain_set's multi_logloss: 0.33947\ttrain_set's balanced_accuracy: 0.987722\tvalid_set's multi_logloss: 0.864843\tvalid_set's balanced_accuracy: 0.496667\n",
      "[62]\ttrain_set's multi_logloss: 0.334466\ttrain_set's balanced_accuracy: 0.987722\tvalid_set's multi_logloss: 0.867278\tvalid_set's balanced_accuracy: 0.508889\n",
      "[63]\ttrain_set's multi_logloss: 0.33058\ttrain_set's balanced_accuracy: 0.987722\tvalid_set's multi_logloss: 0.867354\tvalid_set's balanced_accuracy: 0.508889\n",
      "[64]\ttrain_set's multi_logloss: 0.32549\ttrain_set's balanced_accuracy: 0.987722\tvalid_set's multi_logloss: 0.868367\tvalid_set's balanced_accuracy: 0.515556\n",
      "[65]\ttrain_set's multi_logloss: 0.321877\ttrain_set's balanced_accuracy: 0.9891\tvalid_set's multi_logloss: 0.870384\tvalid_set's balanced_accuracy: 0.508889\n",
      "[66]\ttrain_set's multi_logloss: 0.316911\ttrain_set's balanced_accuracy: 0.990792\tvalid_set's multi_logloss: 0.873369\tvalid_set's balanced_accuracy: 0.508889\n",
      "[67]\ttrain_set's multi_logloss: 0.312941\ttrain_set's balanced_accuracy: 0.9891\tvalid_set's multi_logloss: 0.874593\tvalid_set's balanced_accuracy: 0.503333\n",
      "[68]\ttrain_set's multi_logloss: 0.308681\ttrain_set's balanced_accuracy: 0.990792\tvalid_set's multi_logloss: 0.873656\tvalid_set's balanced_accuracy: 0.508889\n",
      "[69]\ttrain_set's multi_logloss: 0.304705\ttrain_set's balanced_accuracy: 0.992169\tvalid_set's multi_logloss: 0.873497\tvalid_set's balanced_accuracy: 0.508889\n",
      "[70]\ttrain_set's multi_logloss: 0.299989\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.875977\tvalid_set's balanced_accuracy: 0.503333\n",
      "[71]\ttrain_set's multi_logloss: 0.296129\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.875359\tvalid_set's balanced_accuracy: 0.508889\n",
      "[72]\ttrain_set's multi_logloss: 0.292486\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.876354\tvalid_set's balanced_accuracy: 0.508889\n",
      "[73]\ttrain_set's multi_logloss: 0.288496\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.87529\tvalid_set's balanced_accuracy: 0.508889\n",
      "[74]\ttrain_set's multi_logloss: 0.284679\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.875987\tvalid_set's balanced_accuracy: 0.514444\n",
      "[75]\ttrain_set's multi_logloss: 0.280502\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.877004\tvalid_set's balanced_accuracy: 0.508889\n",
      "[76]\ttrain_set's multi_logloss: 0.277158\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.87783\tvalid_set's balanced_accuracy: 0.508889\n",
      "[77]\ttrain_set's multi_logloss: 0.273004\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.878892\tvalid_set's balanced_accuracy: 0.508889\n",
      "[78]\ttrain_set's multi_logloss: 0.269314\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.880505\tvalid_set's balanced_accuracy: 0.508889\n",
      "[79]\ttrain_set's multi_logloss: 0.266404\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.882847\tvalid_set's balanced_accuracy: 0.514444\n",
      "[80]\ttrain_set's multi_logloss: 0.262942\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.88643\tvalid_set's balanced_accuracy: 0.514444\n",
      "[81]\ttrain_set's multi_logloss: 0.259653\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.885262\tvalid_set's balanced_accuracy: 0.52\n",
      "[82]\ttrain_set's multi_logloss: 0.255754\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.886204\tvalid_set's balanced_accuracy: 0.507778\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[83]\ttrain_set's multi_logloss: 0.253067\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.887874\tvalid_set's balanced_accuracy: 0.52\n",
      "[84]\ttrain_set's multi_logloss: 0.249826\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.889205\tvalid_set's balanced_accuracy: 0.52\n",
      "[85]\ttrain_set's multi_logloss: 0.246746\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.887566\tvalid_set's balanced_accuracy: 0.52\n",
      "[86]\ttrain_set's multi_logloss: 0.243321\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.889642\tvalid_set's balanced_accuracy: 0.513333\n",
      "[87]\ttrain_set's multi_logloss: 0.240115\ttrain_set's balanced_accuracy: 0.993546\tvalid_set's multi_logloss: 0.89414\tvalid_set's balanced_accuracy: 0.507778\n",
      "[88]\ttrain_set's multi_logloss: 0.236816\ttrain_set's balanced_accuracy: 0.995238\tvalid_set's multi_logloss: 0.892279\tvalid_set's balanced_accuracy: 0.507778\n",
      "[89]\ttrain_set's multi_logloss: 0.233681\ttrain_set's balanced_accuracy: 0.995238\tvalid_set's multi_logloss: 0.894577\tvalid_set's balanced_accuracy: 0.508889\n",
      "[90]\ttrain_set's multi_logloss: 0.230394\ttrain_set's balanced_accuracy: 0.995238\tvalid_set's multi_logloss: 0.894406\tvalid_set's balanced_accuracy: 0.514444\n",
      "[91]\ttrain_set's multi_logloss: 0.227139\ttrain_set's balanced_accuracy: 0.996931\tvalid_set's multi_logloss: 0.892394\tvalid_set's balanced_accuracy: 0.514444\n",
      "[92]\ttrain_set's multi_logloss: 0.224695\ttrain_set's balanced_accuracy: 0.996931\tvalid_set's multi_logloss: 0.890682\tvalid_set's balanced_accuracy: 0.507778\n",
      "[93]\ttrain_set's multi_logloss: 0.221865\ttrain_set's balanced_accuracy: 0.996931\tvalid_set's multi_logloss: 0.891667\tvalid_set's balanced_accuracy: 0.507778\n",
      "[94]\ttrain_set's multi_logloss: 0.218749\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.891723\tvalid_set's balanced_accuracy: 0.507778\n",
      "[95]\ttrain_set's multi_logloss: 0.216219\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.893345\tvalid_set's balanced_accuracy: 0.501111\n",
      "[96]\ttrain_set's multi_logloss: 0.213659\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.894294\tvalid_set's balanced_accuracy: 0.502222\n",
      "[97]\ttrain_set's multi_logloss: 0.21037\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.896374\tvalid_set's balanced_accuracy: 0.496667\n",
      "[98]\ttrain_set's multi_logloss: 0.207685\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.899075\tvalid_set's balanced_accuracy: 0.508889\n",
      "[99]\ttrain_set's multi_logloss: 0.204735\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.897913\tvalid_set's balanced_accuracy: 0.495556\n",
      "[100]\ttrain_set's multi_logloss: 0.202265\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.899366\tvalid_set's balanced_accuracy: 0.49\n",
      "[101]\ttrain_set's multi_logloss: 0.199854\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.902474\tvalid_set's balanced_accuracy: 0.488889\n",
      "[102]\ttrain_set's multi_logloss: 0.197506\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.905218\tvalid_set's balanced_accuracy: 0.483333\n",
      "[103]\ttrain_set's multi_logloss: 0.195053\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.907278\tvalid_set's balanced_accuracy: 0.483333\n",
      "[104]\ttrain_set's multi_logloss: 0.19272\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.905636\tvalid_set's balanced_accuracy: 0.483333\n",
      "[105]\ttrain_set's multi_logloss: 0.19046\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.906634\tvalid_set's balanced_accuracy: 0.483333\n",
      "[106]\ttrain_set's multi_logloss: 0.188174\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.911063\tvalid_set's balanced_accuracy: 0.483333\n",
      "[107]\ttrain_set's multi_logloss: 0.18606\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.913494\tvalid_set's balanced_accuracy: 0.483333\n",
      "[108]\ttrain_set's multi_logloss: 0.183436\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.91405\tvalid_set's balanced_accuracy: 0.49\n",
      "[109]\ttrain_set's multi_logloss: 0.181223\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.913317\tvalid_set's balanced_accuracy: 0.496667\n",
      "[110]\ttrain_set's multi_logloss: 0.178884\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.912253\tvalid_set's balanced_accuracy: 0.483333\n",
      "[111]\ttrain_set's multi_logloss: 0.17667\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.91324\tvalid_set's balanced_accuracy: 0.483333\n",
      "[112]\ttrain_set's multi_logloss: 0.174549\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.91577\tvalid_set's balanced_accuracy: 0.483333\n",
      "[113]\ttrain_set's multi_logloss: 0.172273\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.915504\tvalid_set's balanced_accuracy: 0.483333\n",
      "[114]\ttrain_set's multi_logloss: 0.170055\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.916141\tvalid_set's balanced_accuracy: 0.483333\n",
      "[115]\ttrain_set's multi_logloss: 0.168098\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.916341\tvalid_set's balanced_accuracy: 0.483333\n",
      "[116]\ttrain_set's multi_logloss: 0.165876\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.917419\tvalid_set's balanced_accuracy: 0.483333\n",
      "[117]\ttrain_set's multi_logloss: 0.163976\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.919944\tvalid_set's balanced_accuracy: 0.483333\n",
      "[118]\ttrain_set's multi_logloss: 0.16202\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.923145\tvalid_set's balanced_accuracy: 0.483333\n",
      "[119]\ttrain_set's multi_logloss: 0.160126\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.925406\tvalid_set's balanced_accuracy: 0.483333\n",
      "[120]\ttrain_set's multi_logloss: 0.15829\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.929828\tvalid_set's balanced_accuracy: 0.483333\n",
      "[121]\ttrain_set's multi_logloss: 0.156438\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.932895\tvalid_set's balanced_accuracy: 0.483333\n",
      "[122]\ttrain_set's multi_logloss: 0.154473\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.933357\tvalid_set's balanced_accuracy: 0.483333\n",
      "[123]\ttrain_set's multi_logloss: 0.152598\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.936829\tvalid_set's balanced_accuracy: 0.483333\n",
      "[124]\ttrain_set's multi_logloss: 0.150727\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.941342\tvalid_set's balanced_accuracy: 0.483333\n",
      "[125]\ttrain_set's multi_logloss: 0.149088\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.943353\tvalid_set's balanced_accuracy: 0.45\n",
      "[126]\ttrain_set's multi_logloss: 0.147405\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.942181\tvalid_set's balanced_accuracy: 0.45\n",
      "[127]\ttrain_set's multi_logloss: 0.145729\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.943704\tvalid_set's balanced_accuracy: 0.45\n",
      "[128]\ttrain_set's multi_logloss: 0.144194\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.947188\tvalid_set's balanced_accuracy: 0.45\n",
      "[129]\ttrain_set's multi_logloss: 0.142339\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.949367\tvalid_set's balanced_accuracy: 0.45\n",
      "[130]\ttrain_set's multi_logloss: 0.140736\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.952057\tvalid_set's balanced_accuracy: 0.45\n",
      "[131]\ttrain_set's multi_logloss: 0.138993\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.951871\tvalid_set's balanced_accuracy: 0.456667\n",
      "[132]\ttrain_set's multi_logloss: 0.137367\ttrain_set's balanced_accuracy: 0.998623\tvalid_set's multi_logloss: 0.954244\tvalid_set's balanced_accuracy: 0.456667\n",
      "[133]\ttrain_set's multi_logloss: 0.135805\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.955854\tvalid_set's balanced_accuracy: 0.456667\n",
      "[134]\ttrain_set's multi_logloss: 0.134056\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.957035\tvalid_set's balanced_accuracy: 0.456667\n",
      "[135]\ttrain_set's multi_logloss: 0.132464\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.957871\tvalid_set's balanced_accuracy: 0.463333\n",
      "[136]\ttrain_set's multi_logloss: 0.130833\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.958805\tvalid_set's balanced_accuracy: 0.463333\n",
      "[137]\ttrain_set's multi_logloss: 0.129614\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.960389\tvalid_set's balanced_accuracy: 0.456667\n",
      "[138]\ttrain_set's multi_logloss: 0.127963\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.962512\tvalid_set's balanced_accuracy: 0.443333\n",
      "[139]\ttrain_set's multi_logloss: 0.126425\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.963114\tvalid_set's balanced_accuracy: 0.45\n",
      "[140]\ttrain_set's multi_logloss: 0.124995\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.964772\tvalid_set's balanced_accuracy: 0.483333\n",
      "[141]\ttrain_set's multi_logloss: 0.123464\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.966164\tvalid_set's balanced_accuracy: 0.483333\n",
      "[142]\ttrain_set's multi_logloss: 0.122165\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.967313\tvalid_set's balanced_accuracy: 0.483333\n",
      "[143]\ttrain_set's multi_logloss: 0.120985\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.971789\tvalid_set's balanced_accuracy: 0.483333\n",
      "[144]\ttrain_set's multi_logloss: 0.119649\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.972248\tvalid_set's balanced_accuracy: 0.477778\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[145]\ttrain_set's multi_logloss: 0.118213\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.972034\tvalid_set's balanced_accuracy: 0.471111\n",
      "[146]\ttrain_set's multi_logloss: 0.116672\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.971669\tvalid_set's balanced_accuracy: 0.476667\n",
      "[147]\ttrain_set's multi_logloss: 0.115448\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.970194\tvalid_set's balanced_accuracy: 0.483333\n",
      "[148]\ttrain_set's multi_logloss: 0.113941\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.969214\tvalid_set's balanced_accuracy: 0.483333\n",
      "[149]\ttrain_set's multi_logloss: 0.112803\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.969048\tvalid_set's balanced_accuracy: 0.476667\n",
      "[150]\ttrain_set's multi_logloss: 0.111511\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.970554\tvalid_set's balanced_accuracy: 0.483333\n",
      "[151]\ttrain_set's multi_logloss: 0.110035\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.974379\tvalid_set's balanced_accuracy: 0.483333\n",
      "[152]\ttrain_set's multi_logloss: 0.108854\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.973756\tvalid_set's balanced_accuracy: 0.476667\n",
      "[153]\ttrain_set's multi_logloss: 0.107786\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.97478\tvalid_set's balanced_accuracy: 0.476667\n",
      "[154]\ttrain_set's multi_logloss: 0.106473\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.974605\tvalid_set's balanced_accuracy: 0.476667\n",
      "[155]\ttrain_set's multi_logloss: 0.105281\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.975044\tvalid_set's balanced_accuracy: 0.476667\n",
      "[156]\ttrain_set's multi_logloss: 0.104083\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.97852\tvalid_set's balanced_accuracy: 0.476667\n",
      "[157]\ttrain_set's multi_logloss: 0.102974\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.981764\tvalid_set's balanced_accuracy: 0.476667\n",
      "[158]\ttrain_set's multi_logloss: 0.101794\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.984295\tvalid_set's balanced_accuracy: 0.476667\n",
      "[159]\ttrain_set's multi_logloss: 0.100795\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.984669\tvalid_set's balanced_accuracy: 0.476667\n",
      "[160]\ttrain_set's multi_logloss: 0.0995258\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.986507\tvalid_set's balanced_accuracy: 0.476667\n",
      "[161]\ttrain_set's multi_logloss: 0.0984467\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.988322\tvalid_set's balanced_accuracy: 0.476667\n",
      "[162]\ttrain_set's multi_logloss: 0.0972898\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.989602\tvalid_set's balanced_accuracy: 0.476667\n",
      "[163]\ttrain_set's multi_logloss: 0.096147\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.989173\tvalid_set's balanced_accuracy: 0.476667\n",
      "[164]\ttrain_set's multi_logloss: 0.0948998\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.991437\tvalid_set's balanced_accuracy: 0.476667\n",
      "[165]\ttrain_set's multi_logloss: 0.0938026\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.994343\tvalid_set's balanced_accuracy: 0.476667\n",
      "[166]\ttrain_set's multi_logloss: 0.0926939\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.996207\tvalid_set's balanced_accuracy: 0.476667\n",
      "[167]\ttrain_set's multi_logloss: 0.0916687\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.995963\tvalid_set's balanced_accuracy: 0.476667\n",
      "[168]\ttrain_set's multi_logloss: 0.0907528\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.998296\tvalid_set's balanced_accuracy: 0.476667\n",
      "[169]\ttrain_set's multi_logloss: 0.0894463\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.998708\tvalid_set's balanced_accuracy: 0.476667\n",
      "[170]\ttrain_set's multi_logloss: 0.0885487\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00028\tvalid_set's balanced_accuracy: 0.476667\n",
      "[171]\ttrain_set's multi_logloss: 0.0876891\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00218\tvalid_set's balanced_accuracy: 0.476667\n",
      "[172]\ttrain_set's multi_logloss: 0.0865047\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00266\tvalid_set's balanced_accuracy: 0.476667\n",
      "[173]\ttrain_set's multi_logloss: 0.0854814\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00411\tvalid_set's balanced_accuracy: 0.476667\n",
      "[174]\ttrain_set's multi_logloss: 0.0845031\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0052\tvalid_set's balanced_accuracy: 0.47\n",
      "[175]\ttrain_set's multi_logloss: 0.0836199\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00288\tvalid_set's balanced_accuracy: 0.47\n",
      "[176]\ttrain_set's multi_logloss: 0.082563\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00553\tvalid_set's balanced_accuracy: 0.47\n",
      "[177]\ttrain_set's multi_logloss: 0.0817061\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00598\tvalid_set's balanced_accuracy: 0.47\n",
      "[178]\ttrain_set's multi_logloss: 0.0807836\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0065\tvalid_set's balanced_accuracy: 0.47\n",
      "[179]\ttrain_set's multi_logloss: 0.0799158\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00861\tvalid_set's balanced_accuracy: 0.47\n",
      "[180]\ttrain_set's multi_logloss: 0.0790278\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01166\tvalid_set's balanced_accuracy: 0.47\n",
      "[181]\ttrain_set's multi_logloss: 0.0781676\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01374\tvalid_set's balanced_accuracy: 0.47\n",
      "[182]\ttrain_set's multi_logloss: 0.0773599\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01458\tvalid_set's balanced_accuracy: 0.47\n",
      "[183]\ttrain_set's multi_logloss: 0.0764366\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01623\tvalid_set's balanced_accuracy: 0.47\n",
      "[184]\ttrain_set's multi_logloss: 0.0755931\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01888\tvalid_set's balanced_accuracy: 0.47\n",
      "[185]\ttrain_set's multi_logloss: 0.074784\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02023\tvalid_set's balanced_accuracy: 0.47\n",
      "[186]\ttrain_set's multi_logloss: 0.0739991\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02065\tvalid_set's balanced_accuracy: 0.47\n",
      "[187]\ttrain_set's multi_logloss: 0.0732784\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02109\tvalid_set's balanced_accuracy: 0.47\n",
      "[188]\ttrain_set's multi_logloss: 0.0723472\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02317\tvalid_set's balanced_accuracy: 0.47\n",
      "[189]\ttrain_set's multi_logloss: 0.0712876\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02799\tvalid_set's balanced_accuracy: 0.47\n",
      "[190]\ttrain_set's multi_logloss: 0.0704996\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02931\tvalid_set's balanced_accuracy: 0.47\n",
      "[191]\ttrain_set's multi_logloss: 0.0698179\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02862\tvalid_set's balanced_accuracy: 0.47\n",
      "[192]\ttrain_set's multi_logloss: 0.0690188\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03137\tvalid_set's balanced_accuracy: 0.47\n",
      "[193]\ttrain_set's multi_logloss: 0.068141\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0323\tvalid_set's balanced_accuracy: 0.47\n",
      "[194]\ttrain_set's multi_logloss: 0.0674825\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03446\tvalid_set's balanced_accuracy: 0.47\n",
      "[195]\ttrain_set's multi_logloss: 0.0666652\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03631\tvalid_set's balanced_accuracy: 0.47\n",
      "[196]\ttrain_set's multi_logloss: 0.0658302\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03806\tvalid_set's balanced_accuracy: 0.47\n",
      "[197]\ttrain_set's multi_logloss: 0.0651083\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04147\tvalid_set's balanced_accuracy: 0.47\n",
      "[198]\ttrain_set's multi_logloss: 0.0644506\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04091\tvalid_set's balanced_accuracy: 0.47\n",
      "[199]\ttrain_set's multi_logloss: 0.0636638\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04373\tvalid_set's balanced_accuracy: 0.47\n",
      "[200]\ttrain_set's multi_logloss: 0.0630397\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04507\tvalid_set's balanced_accuracy: 0.47\n",
      "[201]\ttrain_set's multi_logloss: 0.0623276\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04774\tvalid_set's balanced_accuracy: 0.47\n",
      "[202]\ttrain_set's multi_logloss: 0.061698\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04956\tvalid_set's balanced_accuracy: 0.47\n",
      "[203]\ttrain_set's multi_logloss: 0.061057\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04968\tvalid_set's balanced_accuracy: 0.47\n",
      "[204]\ttrain_set's multi_logloss: 0.0603326\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04998\tvalid_set's balanced_accuracy: 0.47\n",
      "[205]\ttrain_set's multi_logloss: 0.0597321\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04983\tvalid_set's balanced_accuracy: 0.47\n",
      "[206]\ttrain_set's multi_logloss: 0.0591497\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05052\tvalid_set's balanced_accuracy: 0.47\n",
      "[207]\ttrain_set's multi_logloss: 0.0583924\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05238\tvalid_set's balanced_accuracy: 0.47\n",
      "[208]\ttrain_set's multi_logloss: 0.0577922\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05319\tvalid_set's balanced_accuracy: 0.463333\n",
      "[209]\ttrain_set's multi_logloss: 0.0571694\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05429\tvalid_set's balanced_accuracy: 0.47\n",
      "[210]\ttrain_set's multi_logloss: 0.0565936\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05614\tvalid_set's balanced_accuracy: 0.463333\n",
      "[211]\ttrain_set's multi_logloss: 0.0560094\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05901\tvalid_set's balanced_accuracy: 0.463333\n",
      "[212]\ttrain_set's multi_logloss: 0.0554331\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05962\tvalid_set's balanced_accuracy: 0.463333\n",
      "[213]\ttrain_set's multi_logloss: 0.0549258\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06072\tvalid_set's balanced_accuracy: 0.463333\n",
      "[214]\ttrain_set's multi_logloss: 0.0542868\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06262\tvalid_set's balanced_accuracy: 0.463333\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200529/models/LightGBMClassifier/model.pkl\n",
      "\t0.52\t = Validation balanced_accuracy score\n",
      "\t0.84s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: CatboostClassifier ...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[215]\ttrain_set's multi_logloss: 0.0537028\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0632\tvalid_set's balanced_accuracy: 0.47\n",
      "[216]\ttrain_set's multi_logloss: 0.0531629\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06127\tvalid_set's balanced_accuracy: 0.47\n",
      "[217]\ttrain_set's multi_logloss: 0.0525888\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06063\tvalid_set's balanced_accuracy: 0.47\n",
      "[218]\ttrain_set's multi_logloss: 0.0520365\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06212\tvalid_set's balanced_accuracy: 0.47\n",
      "[219]\ttrain_set's multi_logloss: 0.0514125\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06323\tvalid_set's balanced_accuracy: 0.47\n",
      "[220]\ttrain_set's multi_logloss: 0.050813\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06389\tvalid_set's balanced_accuracy: 0.47\n",
      "[221]\ttrain_set's multi_logloss: 0.0503173\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0645\tvalid_set's balanced_accuracy: 0.47\n",
      "[222]\ttrain_set's multi_logloss: 0.0497348\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06526\tvalid_set's balanced_accuracy: 0.47\n",
      "[223]\ttrain_set's multi_logloss: 0.0491566\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06574\tvalid_set's balanced_accuracy: 0.47\n",
      "[224]\ttrain_set's multi_logloss: 0.0486902\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06752\tvalid_set's balanced_accuracy: 0.47\n",
      "[225]\ttrain_set's multi_logloss: 0.0481102\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07027\tvalid_set's balanced_accuracy: 0.47\n",
      "[226]\ttrain_set's multi_logloss: 0.0476111\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07038\tvalid_set's balanced_accuracy: 0.47\n",
      "[227]\ttrain_set's multi_logloss: 0.0470997\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07283\tvalid_set's balanced_accuracy: 0.47\n",
      "[228]\ttrain_set's multi_logloss: 0.0466768\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07357\tvalid_set's balanced_accuracy: 0.47\n",
      "[229]\ttrain_set's multi_logloss: 0.0462129\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07562\tvalid_set's balanced_accuracy: 0.47\n",
      "[230]\ttrain_set's multi_logloss: 0.0456371\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07879\tvalid_set's balanced_accuracy: 0.47\n",
      "[231]\ttrain_set's multi_logloss: 0.0451897\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.08094\tvalid_set's balanced_accuracy: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': <autogluon.utils.tabular.ml.models.catboost.catboost_utils.MulticlassCustomMetric object at 0x11c1170d0>}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.3682829\ttest: 0.3344444\tbest: 0.3344444 (0)\ttotal: 109ms\tremaining: 18m 14s\n",
      "1:\tlearn: 0.3509600\ttest: 0.3377778\tbest: 0.3377778 (1)\ttotal: 129ms\tremaining: 10m 46s\n",
      "2:\tlearn: 0.3427375\ttest: 0.3466667\tbest: 0.3466667 (2)\ttotal: 136ms\tremaining: 7m 34s\n",
      "3:\tlearn: 0.3387241\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 140ms\tremaining: 5m 50s\n",
      "4:\tlearn: 0.3401015\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 147ms\tremaining: 4m 54s\n",
      "5:\tlearn: 0.3451777\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 170ms\tremaining: 4m 43s\n",
      "6:\tlearn: 0.3417936\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 210ms\tremaining: 4m 59s\n",
      "7:\tlearn: 0.3502538\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 240ms\tremaining: 4m 59s\n",
      "8:\tlearn: 0.3502538\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 246ms\tremaining: 4m 32s\n",
      "9:\tlearn: 0.3502538\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 261ms\tremaining: 4m 20s\n",
      "10:\tlearn: 0.3519459\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 290ms\tremaining: 4m 23s\n",
      "11:\tlearn: 0.3539525\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 324ms\tremaining: 4m 29s\n",
      "12:\tlearn: 0.3556446\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 374ms\tremaining: 4m 47s\n",
      "13:\tlearn: 0.3488764\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 413ms\tremaining: 4m 54s\n",
      "14:\tlearn: 0.3624128\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 441ms\tremaining: 4m 53s\n",
      "15:\tlearn: 0.3674889\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 476ms\tremaining: 4m 56s\n",
      "16:\tlearn: 0.3705584\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 511ms\tremaining: 5m\n",
      "17:\tlearn: 0.3722504\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 535ms\tremaining: 4m 56s\n",
      "18:\tlearn: 0.3773266\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 573ms\tremaining: 5m\n",
      "19:\tlearn: 0.3824027\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 593ms\tremaining: 4m 56s\n",
      "20:\tlearn: 0.3840948\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 618ms\tremaining: 4m 53s\n",
      "21:\tlearn: 0.3840948\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 627ms\tremaining: 4m 44s\n",
      "22:\tlearn: 0.3844094\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 662ms\tremaining: 4m 47s\n",
      "23:\tlearn: 0.3901148\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 699ms\tremaining: 4m 50s\n",
      "24:\tlearn: 0.3881081\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 738ms\tremaining: 4m 54s\n",
      "25:\tlearn: 0.3918069\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 768ms\tremaining: 4m 54s\n",
      "26:\tlearn: 0.3901148\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 788ms\tremaining: 4m 51s\n",
      "27:\tlearn: 0.3951909\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 810ms\tremaining: 4m 48s\n",
      "28:\tlearn: 0.3968830\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 849ms\tremaining: 4m 51s\n",
      "29:\tlearn: 0.4005817\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 870ms\tremaining: 4m 48s\n",
      "30:\tlearn: 0.3988897\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 896ms\tremaining: 4m 48s\n",
      "31:\tlearn: 0.4036512\ttest: 0.3333333\tbest: 0.3466667 (2)\ttotal: 918ms\tremaining: 4m 45s\n",
      "32:\tlearn: 0.4039658\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 939ms\tremaining: 4m 43s\n",
      "33:\tlearn: 0.4056579\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 962ms\tremaining: 4m 41s\n",
      "34:\tlearn: 0.4093566\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 1s\tremaining: 4m 44s\n",
      "35:\tlearn: 0.4205717\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.02s\tremaining: 4m 42s\n",
      "36:\tlearn: 0.4208863\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.04s\tremaining: 4m 40s\n",
      "37:\tlearn: 0.4293465\ttest: 0.3277778\tbest: 0.3466667 (2)\ttotal: 1.07s\tremaining: 4m 40s\n",
      "38:\tlearn: 0.4327306\ttest: 0.3344444\tbest: 0.3466667 (2)\ttotal: 1.09s\tremaining: 4m 38s\n",
      "39:\tlearn: 0.4316679\ttest: 0.3344444\tbest: 0.3466667 (2)\ttotal: 1.13s\tremaining: 4m 40s\n",
      "40:\tlearn: 0.4339892\ttest: 0.3288889\tbest: 0.3466667 (2)\ttotal: 1.14s\tremaining: 4m 37s\n",
      "41:\tlearn: 0.4346185\ttest: 0.3277778\tbest: 0.3466667 (2)\ttotal: 1.2s\tremaining: 4m 43s\n",
      "42:\tlearn: 0.4430787\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.23s\tremaining: 4m 44s\n",
      "43:\tlearn: 0.4467774\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.27s\tremaining: 4m 46s\n",
      "44:\tlearn: 0.4590553\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.3s\tremaining: 4m 47s\n",
      "45:\tlearn: 0.4522871\ttest: 0.3222222\tbest: 0.3466667 (2)\ttotal: 1.33s\tremaining: 4m 47s\n",
      "46:\tlearn: 0.4579925\ttest: 0.3233333\tbest: 0.3466667 (2)\ttotal: 1.39s\tremaining: 4m 53s\n",
      "47:\tlearn: 0.4607473\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 1.42s\tremaining: 4m 53s\n",
      "48:\tlearn: 0.4624393\ttest: 0.3400000\tbest: 0.3466667 (2)\ttotal: 1.46s\tremaining: 4m 56s\n",
      "49:\tlearn: 0.4627540\ttest: 0.3411111\tbest: 0.3466667 (2)\ttotal: 1.49s\tremaining: 4m 55s\n",
      "50:\tlearn: 0.4647607\ttest: 0.3422222\tbest: 0.3466667 (2)\ttotal: 1.52s\tremaining: 4m 57s\n",
      "51:\tlearn: 0.4647607\ttest: 0.3422222\tbest: 0.3466667 (2)\ttotal: 1.55s\tremaining: 4m 56s\n",
      "52:\tlearn: 0.4779669\ttest: 0.3477778\tbest: 0.3477778 (52)\ttotal: 1.57s\tremaining: 4m 55s\n",
      "53:\tlearn: 0.4762748\ttest: 0.3477778\tbest: 0.3477778 (52)\ttotal: 1.61s\tremaining: 4m 57s\n",
      "54:\tlearn: 0.4785962\ttest: 0.3377778\tbest: 0.3477778 (52)\ttotal: 1.65s\tremaining: 4m 58s\n",
      "55:\tlearn: 0.4718435\ttest: 0.3444444\tbest: 0.3477778 (52)\ttotal: 1.69s\tremaining: 4m 59s\n",
      "56:\tlearn: 0.4856790\ttest: 0.3511111\tbest: 0.3511111 (56)\ttotal: 1.72s\tremaining: 4m 59s\n",
      "57:\tlearn: 0.4789263\ttest: 0.3277778\tbest: 0.3511111 (56)\ttotal: 1.75s\tremaining: 4m 59s\n",
      "58:\tlearn: 0.4921325\ttest: 0.3388889\tbest: 0.3511111 (56)\ttotal: 1.78s\tremaining: 5m\n",
      "59:\tlearn: 0.4853799\ttest: 0.3355556\tbest: 0.3511111 (56)\ttotal: 1.81s\tremaining: 5m\n",
      "60:\tlearn: 0.4941392\ttest: 0.3422222\tbest: 0.3511111 (56)\ttotal: 1.85s\tremaining: 5m 1s\n",
      "61:\tlearn: 0.4904560\ttest: 0.3422222\tbest: 0.3511111 (56)\ttotal: 1.89s\tremaining: 5m 2s\n",
      "62:\tlearn: 0.4938401\ttest: 0.3488889\tbest: 0.3511111 (56)\ttotal: 1.91s\tremaining: 5m 1s\n",
      "63:\tlearn: 0.4978535\ttest: 0.3622222\tbest: 0.3622222 (63)\ttotal: 1.95s\tremaining: 5m 3s\n",
      "64:\tlearn: 0.5090530\ttest: 0.3688889\tbest: 0.3688889 (64)\ttotal: 1.99s\tremaining: 5m 4s\n",
      "65:\tlearn: 0.5161358\ttest: 0.3811111\tbest: 0.3811111 (65)\ttotal: 2.02s\tremaining: 5m 4s\n",
      "66:\tlearn: 0.5208973\ttest: 0.3700000\tbest: 0.3811111 (65)\ttotal: 2.05s\tremaining: 5m 4s\n",
      "67:\tlearn: 0.5178279\ttest: 0.3755556\tbest: 0.3811111 (65)\ttotal: 2.07s\tremaining: 5m 2s\n",
      "68:\tlearn: 0.5270363\ttest: 0.3633333\tbest: 0.3811111 (65)\ttotal: 2.11s\tremaining: 5m 4s\n",
      "69:\tlearn: 0.5301057\ttest: 0.3633333\tbest: 0.3811111 (65)\ttotal: 2.15s\tremaining: 5m 4s\n",
      "70:\tlearn: 0.5293576\ttest: 0.3633333\tbest: 0.3811111 (65)\ttotal: 2.17s\tremaining: 5m 3s\n",
      "71:\tlearn: 0.5388806\ttest: 0.3700000\tbest: 0.3811111 (65)\ttotal: 2.21s\tremaining: 5m 4s\n",
      "72:\tlearn: 0.5537789\ttest: 0.3633333\tbest: 0.3811111 (65)\ttotal: 2.24s\tremaining: 5m 4s\n",
      "73:\tlearn: 0.5537789\ttest: 0.3722222\tbest: 0.3811111 (65)\ttotal: 2.27s\tremaining: 5m 4s\n",
      "74:\tlearn: 0.5656232\ttest: 0.3644444\tbest: 0.3811111 (65)\ttotal: 2.31s\tremaining: 5m 5s\n",
      "75:\tlearn: 0.5513542\ttest: 0.3655556\tbest: 0.3811111 (65)\ttotal: 2.33s\tremaining: 5m 4s\n",
      "76:\tlearn: 0.5645604\ttest: 0.3711111\tbest: 0.3811111 (65)\ttotal: 2.4s\tremaining: 5m 9s\n",
      "77:\tlearn: 0.5628684\ttest: 0.3766667\tbest: 0.3811111 (65)\ttotal: 2.43s\tremaining: 5m 8s\n",
      "78:\tlearn: 0.5564303\ttest: 0.3711111\tbest: 0.3811111 (65)\ttotal: 2.47s\tremaining: 5m 9s\n",
      "79:\tlearn: 0.5696366\ttest: 0.3766667\tbest: 0.3811111 (65)\ttotal: 2.5s\tremaining: 5m 10s\n",
      "80:\tlearn: 0.5740834\ttest: 0.3700000\tbest: 0.3811111 (65)\ttotal: 2.54s\tremaining: 5m 10s\n",
      "81:\tlearn: 0.5757755\ttest: 0.3700000\tbest: 0.3811111 (65)\ttotal: 2.57s\tremaining: 5m 10s\n",
      "82:\tlearn: 0.5822135\ttest: 0.3822222\tbest: 0.3822222 (82)\ttotal: 2.6s\tremaining: 5m 10s\n",
      "83:\tlearn: 0.5937277\ttest: 0.3688889\tbest: 0.3822222 (82)\ttotal: 2.63s\tremaining: 5m 10s\n",
      "84:\tlearn: 0.5940423\ttest: 0.3788889\tbest: 0.3822222 (82)\ttotal: 2.66s\tremaining: 5m 10s\n",
      "85:\tlearn: 0.5937277\ttest: 0.3900000\tbest: 0.3900000 (85)\ttotal: 2.7s\tremaining: 5m 10s\n",
      "86:\tlearn: 0.5937277\ttest: 0.3911111\tbest: 0.3911111 (86)\ttotal: 2.73s\tremaining: 5m 11s\n",
      "87:\tlearn: 0.5920356\ttest: 0.3966667\tbest: 0.3966667 (87)\ttotal: 2.77s\tremaining: 5m 11s\n",
      "88:\tlearn: 0.5988038\ttest: 0.3900000\tbest: 0.3966667 (87)\ttotal: 2.79s\tremaining: 5m 11s\n",
      "89:\tlearn: 0.5984892\ttest: 0.3788889\tbest: 0.3966667 (87)\ttotal: 2.83s\tremaining: 5m 11s\n",
      "90:\tlearn: 0.6035653\ttest: 0.3777778\tbest: 0.3966667 (87)\ttotal: 2.87s\tremaining: 5m 12s\n",
      "91:\tlearn: 0.6103335\ttest: 0.3833333\tbest: 0.3966667 (87)\ttotal: 2.9s\tremaining: 5m 12s\n",
      "92:\tlearn: 0.6167716\ttest: 0.3900000\tbest: 0.3966667 (87)\ttotal: 2.93s\tremaining: 5m 12s\n",
      "93:\tlearn: 0.6150795\ttest: 0.3900000\tbest: 0.3966667 (87)\ttotal: 2.96s\tremaining: 5m 12s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94:\tlearn: 0.6235242\ttest: 0.3955556\tbest: 0.3966667 (87)\ttotal: 3s\tremaining: 5m 13s\n",
      "95:\tlearn: 0.6167716\ttest: 0.3955556\tbest: 0.3966667 (87)\ttotal: 3.04s\tremaining: 5m 13s\n",
      "96:\tlearn: 0.6184636\ttest: 0.3966667\tbest: 0.3966667 (87)\ttotal: 3.07s\tremaining: 5m 13s\n",
      "97:\tlearn: 0.6252163\ttest: 0.4077778\tbest: 0.4077778 (97)\ttotal: 3.1s\tremaining: 5m 13s\n",
      "98:\tlearn: 0.6333464\ttest: 0.3966667\tbest: 0.4077778 (97)\ttotal: 3.13s\tremaining: 5m 13s\n",
      "99:\tlearn: 0.6364158\ttest: 0.3955556\tbest: 0.4077778 (97)\ttotal: 3.17s\tremaining: 5m 13s\n",
      "100:\tlearn: 0.6397999\ttest: 0.3877778\tbest: 0.4077778 (97)\ttotal: 3.2s\tremaining: 5m 14s\n",
      "101:\tlearn: 0.6401146\ttest: 0.3888889\tbest: 0.4077778 (97)\ttotal: 3.24s\tremaining: 5m 14s\n",
      "102:\tlearn: 0.6401146\ttest: 0.3944444\tbest: 0.4077778 (97)\ttotal: 3.29s\tremaining: 5m 16s\n",
      "103:\tlearn: 0.6418066\ttest: 0.4011111\tbest: 0.4077778 (97)\ttotal: 3.33s\tremaining: 5m 17s\n",
      "104:\tlearn: 0.6434986\ttest: 0.4077778\tbest: 0.4077778 (97)\ttotal: 3.37s\tremaining: 5m 17s\n",
      "105:\tlearn: 0.6431840\ttest: 0.4022222\tbest: 0.4077778 (97)\ttotal: 3.4s\tremaining: 5m 17s\n",
      "106:\tlearn: 0.6418066\ttest: 0.4088889\tbest: 0.4088889 (106)\ttotal: 3.45s\tremaining: 5m 18s\n",
      "107:\tlearn: 0.6488739\ttest: 0.4077778\tbest: 0.4088889 (106)\ttotal: 3.47s\tremaining: 5m 17s\n",
      "108:\tlearn: 0.6580668\ttest: 0.4022222\tbest: 0.4088889 (106)\ttotal: 3.51s\tremaining: 5m 18s\n",
      "109:\tlearn: 0.6549973\ttest: 0.4077778\tbest: 0.4088889 (106)\ttotal: 3.54s\tremaining: 5m 18s\n",
      "110:\tlearn: 0.6651341\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 3.57s\tremaining: 5m 18s\n",
      "111:\tlearn: 0.6661968\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 3.59s\tremaining: 5m 17s\n",
      "112:\tlearn: 0.6580668\ttest: 0.4022222\tbest: 0.4088889 (106)\ttotal: 3.63s\tremaining: 5m 17s\n",
      "113:\tlearn: 0.6583814\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 3.67s\tremaining: 5m 18s\n",
      "114:\tlearn: 0.6566894\ttest: 0.3911111\tbest: 0.4088889 (106)\ttotal: 3.7s\tremaining: 5m 18s\n",
      "115:\tlearn: 0.6628283\ttest: 0.3855556\tbest: 0.4088889 (106)\ttotal: 3.74s\tremaining: 5m 18s\n",
      "116:\tlearn: 0.6889261\ttest: 0.3988889\tbest: 0.4088889 (106)\ttotal: 3.77s\tremaining: 5m 18s\n",
      "117:\tlearn: 0.6875487\ttest: 0.3988889\tbest: 0.4088889 (106)\ttotal: 3.81s\tremaining: 5m 18s\n",
      "118:\tlearn: 0.6956787\ttest: 0.3988889\tbest: 0.4088889 (106)\ttotal: 3.85s\tremaining: 5m 19s\n",
      "119:\tlearn: 0.6990628\ttest: 0.3866667\tbest: 0.4088889 (106)\ttotal: 3.88s\tremaining: 5m 19s\n",
      "120:\tlearn: 0.6990628\ttest: 0.3866667\tbest: 0.4088889 (106)\ttotal: 3.93s\tremaining: 5m 20s\n",
      "121:\tlearn: 0.7041390\ttest: 0.3866667\tbest: 0.4088889 (106)\ttotal: 3.98s\tremaining: 5m 22s\n",
      "122:\tlearn: 0.7122691\ttest: 0.3866667\tbest: 0.4088889 (106)\ttotal: 4.03s\tremaining: 5m 23s\n",
      "123:\tlearn: 0.7105770\ttest: 0.3866667\tbest: 0.4088889 (106)\ttotal: 4.05s\tremaining: 5m 22s\n",
      "124:\tlearn: 0.7091996\ttest: 0.3877778\tbest: 0.4088889 (106)\ttotal: 4.08s\tremaining: 5m 22s\n",
      "125:\tlearn: 0.7122691\ttest: 0.3811111\tbest: 0.4088889 (106)\ttotal: 4.11s\tremaining: 5m 22s\n",
      "126:\tlearn: 0.7142758\ttest: 0.3944444\tbest: 0.4088889 (106)\ttotal: 4.14s\tremaining: 5m 21s\n",
      "127:\tlearn: 0.7159678\ttest: 0.3933333\tbest: 0.4088889 (106)\ttotal: 4.18s\tremaining: 5m 22s\n",
      "128:\tlearn: 0.7240979\ttest: 0.3811111\tbest: 0.4088889 (106)\ttotal: 4.21s\tremaining: 5m 21s\n",
      "129:\tlearn: 0.7193519\ttest: 0.3944444\tbest: 0.4088889 (106)\ttotal: 4.24s\tremaining: 5m 21s\n",
      "130:\tlearn: 0.7369895\ttest: 0.3944444\tbest: 0.4088889 (106)\ttotal: 4.28s\tremaining: 5m 22s\n",
      "131:\tlearn: 0.7369895\ttest: 0.3955556\tbest: 0.4088889 (106)\ttotal: 4.31s\tremaining: 5m 22s\n",
      "132:\tlearn: 0.7366748\ttest: 0.4022222\tbest: 0.4088889 (106)\ttotal: 4.35s\tremaining: 5m 22s\n",
      "133:\tlearn: 0.7349828\ttest: 0.3955556\tbest: 0.4088889 (106)\ttotal: 4.4s\tremaining: 5m 23s\n",
      "134:\tlearn: 0.7400589\ttest: 0.3955556\tbest: 0.4088889 (106)\ttotal: 4.42s\tremaining: 5m 23s\n",
      "135:\tlearn: 0.7336209\ttest: 0.3955556\tbest: 0.4088889 (106)\ttotal: 4.45s\tremaining: 5m 22s\n",
      "136:\tlearn: 0.7434430\ttest: 0.3900000\tbest: 0.4088889 (106)\ttotal: 4.49s\tremaining: 5m 23s\n",
      "137:\tlearn: 0.7434430\ttest: 0.3900000\tbest: 0.4088889 (106)\ttotal: 4.53s\tremaining: 5m 23s\n",
      "138:\tlearn: 0.7434430\ttest: 0.3900000\tbest: 0.4088889 (106)\ttotal: 4.57s\tremaining: 5m 24s\n",
      "139:\tlearn: 0.7434430\ttest: 0.3900000\tbest: 0.4088889 (106)\ttotal: 4.6s\tremaining: 5m 24s\n",
      "140:\tlearn: 0.7515731\ttest: 0.3900000\tbest: 0.4088889 (106)\ttotal: 4.63s\tremaining: 5m 24s\n",
      "141:\tlearn: 0.7549572\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.66s\tremaining: 5m 23s\n",
      "142:\tlearn: 0.7597187\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.68s\tremaining: 5m 22s\n",
      "143:\tlearn: 0.7594041\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.71s\tremaining: 5m 22s\n",
      "144:\tlearn: 0.7580267\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 4.74s\tremaining: 5m 22s\n",
      "145:\tlearn: 0.7580267\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 4.77s\tremaining: 5m 21s\n",
      "146:\tlearn: 0.7580267\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 4.8s\tremaining: 5m 21s\n",
      "147:\tlearn: 0.7594041\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 4.84s\tremaining: 5m 21s\n",
      "148:\tlearn: 0.7597187\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 4.87s\tremaining: 5m 21s\n",
      "149:\tlearn: 0.7689116\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.9s\tremaining: 5m 21s\n",
      "150:\tlearn: 0.7692262\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.93s\tremaining: 5m 21s\n",
      "151:\tlearn: 0.7706036\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 4.97s\tremaining: 5m 22s\n",
      "152:\tlearn: 0.7770416\ttest: 0.4044444\tbest: 0.4088889 (106)\ttotal: 5.01s\tremaining: 5m 22s\n",
      "153:\tlearn: 0.7854864\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.04s\tremaining: 5m 22s\n",
      "154:\tlearn: 0.7956231\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.08s\tremaining: 5m 22s\n",
      "155:\tlearn: 0.8082001\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.11s\tremaining: 5m 22s\n",
      "156:\tlearn: 0.8078854\ttest: 0.4044444\tbest: 0.4088889 (106)\ttotal: 5.14s\tremaining: 5m 22s\n",
      "157:\tlearn: 0.8095775\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.17s\tremaining: 5m 22s\n",
      "158:\tlearn: 0.8143235\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.2s\tremaining: 5m 22s\n",
      "159:\tlearn: 0.8210761\ttest: 0.4044444\tbest: 0.4088889 (106)\ttotal: 5.25s\tremaining: 5m 22s\n",
      "160:\tlearn: 0.8227682\ttest: 0.4044444\tbest: 0.4088889 (106)\ttotal: 5.28s\tremaining: 5m 22s\n",
      "161:\tlearn: 0.8305836\ttest: 0.4044444\tbest: 0.4088889 (106)\ttotal: 5.3s\tremaining: 5m 21s\n",
      "162:\tlearn: 0.8322757\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.33s\tremaining: 5m 21s\n",
      "163:\tlearn: 0.8292062\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.36s\tremaining: 5m 21s\n",
      "164:\tlearn: 0.8292062\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.4s\tremaining: 5m 21s\n",
      "165:\tlearn: 0.8325903\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.43s\tremaining: 5m 21s\n",
      "166:\tlearn: 0.8308983\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.49s\tremaining: 5m 22s\n",
      "167:\tlearn: 0.8308983\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.53s\tremaining: 5m 23s\n",
      "168:\tlearn: 0.8342824\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 5.56s\tremaining: 5m 23s\n",
      "169:\tlearn: 0.8325903\ttest: 0.3977778\tbest: 0.4088889 (106)\ttotal: 5.58s\tremaining: 5m 22s\n",
      "170:\tlearn: 0.8342824\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.62s\tremaining: 5m 22s\n",
      "171:\tlearn: 0.8376665\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.67s\tremaining: 5m 24s\n",
      "172:\tlearn: 0.8376665\ttest: 0.3911111\tbest: 0.4088889 (106)\ttotal: 5.72s\tremaining: 5m 24s\n",
      "173:\tlearn: 0.8362890\ttest: 0.3911111\tbest: 0.4088889 (106)\ttotal: 5.75s\tremaining: 5m 24s\n",
      "174:\tlearn: 0.8345970\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.79s\tremaining: 5m 24s\n",
      "175:\tlearn: 0.8362890\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.81s\tremaining: 5m 24s\n",
      "176:\tlearn: 0.8379811\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.84s\tremaining: 5m 23s\n",
      "177:\tlearn: 0.8410505\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.86s\tremaining: 5m 23s\n",
      "178:\tlearn: 0.8410505\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.89s\tremaining: 5m 23s\n",
      "179:\tlearn: 0.8393585\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.91s\tremaining: 5m 22s\n",
      "180:\tlearn: 0.8424280\ttest: 0.4022222\tbest: 0.4088889 (106)\ttotal: 5.95s\tremaining: 5m 22s\n",
      "181:\tlearn: 0.8407359\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 5.99s\tremaining: 5m 22s\n",
      "182:\tlearn: 0.8519355\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 6.02s\tremaining: 5m 22s\n",
      "183:\tlearn: 0.8620722\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 6.06s\tremaining: 5m 23s\n",
      "184:\tlearn: 0.8702023\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 6.09s\tremaining: 5m 23s\n",
      "185:\tlearn: 0.8505580\ttest: 0.3966667\tbest: 0.4088889 (106)\ttotal: 6.13s\tremaining: 5m 23s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "186:\tlearn: 0.8685103\ttest: 0.4033333\tbest: 0.4088889 (106)\ttotal: 6.17s\tremaining: 5m 23s\n",
      "187:\tlearn: 0.8780177\ttest: 0.4100000\tbest: 0.4100000 (187)\ttotal: 6.19s\tremaining: 5m 23s\n",
      "188:\tlearn: 0.8780177\ttest: 0.4033333\tbest: 0.4100000 (187)\ttotal: 6.23s\tremaining: 5m 23s\n",
      "189:\tlearn: 0.8780177\ttest: 0.3966667\tbest: 0.4100000 (187)\ttotal: 6.25s\tremaining: 5m 22s\n",
      "190:\tlearn: 0.8763257\ttest: 0.3966667\tbest: 0.4100000 (187)\ttotal: 6.28s\tremaining: 5m 22s\n",
      "191:\tlearn: 0.8780177\ttest: 0.3966667\tbest: 0.4100000 (187)\ttotal: 6.32s\tremaining: 5m 22s\n",
      "192:\tlearn: 0.8746336\ttest: 0.4033333\tbest: 0.4100000 (187)\ttotal: 6.35s\tremaining: 5m 22s\n",
      "193:\tlearn: 0.8746336\ttest: 0.3922222\tbest: 0.4100000 (187)\ttotal: 6.39s\tremaining: 5m 22s\n",
      "194:\tlearn: 0.8763257\ttest: 0.3988889\tbest: 0.4100000 (187)\ttotal: 6.41s\tremaining: 5m 22s\n",
      "195:\tlearn: 0.8763257\ttest: 0.4044444\tbest: 0.4100000 (187)\ttotal: 6.44s\tremaining: 5m 22s\n",
      "196:\tlearn: 0.8780177\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.49s\tremaining: 5m 22s\n",
      "197:\tlearn: 0.8780177\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.52s\tremaining: 5m 22s\n",
      "198:\tlearn: 0.8780177\ttest: 0.4111111\tbest: 0.4177778 (196)\ttotal: 6.56s\tremaining: 5m 22s\n",
      "199:\tlearn: 0.8861478\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.59s\tremaining: 5m 23s\n",
      "200:\tlearn: 0.8861478\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.63s\tremaining: 5m 23s\n",
      "201:\tlearn: 0.9041000\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.67s\tremaining: 5m 23s\n",
      "202:\tlearn: 0.9041000\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.7s\tremaining: 5m 23s\n",
      "203:\tlearn: 0.9122301\ttest: 0.4044444\tbest: 0.4177778 (196)\ttotal: 6.75s\tremaining: 5m 23s\n",
      "204:\tlearn: 0.9152996\ttest: 0.4044444\tbest: 0.4177778 (196)\ttotal: 6.79s\tremaining: 5m 24s\n",
      "205:\tlearn: 0.9119155\ttest: 0.4044444\tbest: 0.4177778 (196)\ttotal: 6.83s\tremaining: 5m 24s\n",
      "206:\tlearn: 0.9136075\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.86s\tremaining: 5m 24s\n",
      "207:\tlearn: 0.9119155\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.89s\tremaining: 5m 24s\n",
      "208:\tlearn: 0.9217376\ttest: 0.4111111\tbest: 0.4177778 (196)\ttotal: 6.93s\tremaining: 5m 24s\n",
      "209:\tlearn: 0.9217376\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 6.96s\tremaining: 5m 24s\n",
      "210:\tlearn: 0.9169916\ttest: 0.4177778\tbest: 0.4177778 (196)\ttotal: 7.01s\tremaining: 5m 25s\n",
      "211:\tlearn: 0.9396898\ttest: 0.4111111\tbest: 0.4177778 (196)\ttotal: 7.05s\tremaining: 5m 25s\n",
      "212:\tlearn: 0.9396898\ttest: 0.4111111\tbest: 0.4177778 (196)\ttotal: 7.08s\tremaining: 5m 25s\n",
      "213:\tlearn: 0.9413819\ttest: 0.4244444\tbest: 0.4244444 (213)\ttotal: 7.11s\tremaining: 5m 25s\n",
      "214:\tlearn: 0.9396898\ttest: 0.4177778\tbest: 0.4244444 (213)\ttotal: 7.15s\tremaining: 5m 25s\n",
      "215:\tlearn: 0.9413819\ttest: 0.4300000\tbest: 0.4300000 (215)\ttotal: 7.19s\tremaining: 5m 25s\n",
      "216:\tlearn: 0.9508894\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.23s\tremaining: 5m 25s\n",
      "217:\tlearn: 0.9542735\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.26s\tremaining: 5m 25s\n",
      "218:\tlearn: 0.9512040\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 7.29s\tremaining: 5m 25s\n",
      "219:\tlearn: 0.9495119\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.33s\tremaining: 5m 25s\n",
      "220:\tlearn: 0.9512040\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.39s\tremaining: 5m 26s\n",
      "221:\tlearn: 0.9525814\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.42s\tremaining: 5m 26s\n",
      "222:\tlearn: 0.9512040\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.45s\tremaining: 5m 26s\n",
      "223:\tlearn: 0.9512040\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.47s\tremaining: 5m 26s\n",
      "224:\tlearn: 0.9525814\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.5s\tremaining: 5m 25s\n",
      "225:\tlearn: 0.9525814\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.53s\tremaining: 5m 25s\n",
      "226:\tlearn: 0.9705336\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.56s\tremaining: 5m 25s\n",
      "227:\tlearn: 0.9705336\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.6s\tremaining: 5m 25s\n",
      "228:\tlearn: 0.9705336\ttest: 0.4366667\tbest: 0.4366667 (216)\ttotal: 7.65s\tremaining: 5m 26s\n",
      "229:\tlearn: 0.9705336\ttest: 0.4300000\tbest: 0.4366667 (216)\ttotal: 7.68s\tremaining: 5m 26s\n",
      "230:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.72s\tremaining: 5m 26s\n",
      "231:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.76s\tremaining: 5m 26s\n",
      "232:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.8s\tremaining: 5m 26s\n",
      "233:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.82s\tremaining: 5m 26s\n",
      "234:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.86s\tremaining: 5m 26s\n",
      "235:\tlearn: 0.9705336\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 7.9s\tremaining: 5m 26s\n",
      "236:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.93s\tremaining: 5m 26s\n",
      "237:\tlearn: 0.9705336\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 7.95s\tremaining: 5m 26s\n",
      "238:\tlearn: 0.9722257\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 7.99s\tremaining: 5m 26s\n",
      "239:\tlearn: 0.9803557\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 8.03s\tremaining: 5m 26s\n",
      "240:\tlearn: 0.9739177\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 8.06s\tremaining: 5m 26s\n",
      "241:\tlearn: 0.9722257\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.1s\tremaining: 5m 26s\n",
      "242:\tlearn: 0.9722257\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.13s\tremaining: 5m 26s\n",
      "243:\tlearn: 0.9722257\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.18s\tremaining: 5m 27s\n",
      "244:\tlearn: 0.9739177\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.22s\tremaining: 5m 27s\n",
      "245:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.26s\tremaining: 5m 27s\n",
      "246:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.3s\tremaining: 5m 27s\n",
      "247:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 8.32s\tremaining: 5m 27s\n",
      "248:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.37s\tremaining: 5m 27s\n",
      "249:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.41s\tremaining: 5m 27s\n",
      "250:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.44s\tremaining: 5m 27s\n",
      "251:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.49s\tremaining: 5m 28s\n",
      "252:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.53s\tremaining: 5m 28s\n",
      "253:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.56s\tremaining: 5m 28s\n",
      "254:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.6s\tremaining: 5m 28s\n",
      "255:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.64s\tremaining: 5m 28s\n",
      "256:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.68s\tremaining: 5m 29s\n",
      "257:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.72s\tremaining: 5m 29s\n",
      "258:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.76s\tremaining: 5m 29s\n",
      "259:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 8.83s\tremaining: 5m 30s\n",
      "260:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.87s\tremaining: 5m 30s\n",
      "261:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.9s\tremaining: 5m 30s\n",
      "262:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.94s\tremaining: 5m 31s\n",
      "263:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 8.98s\tremaining: 5m 31s\n",
      "264:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.02s\tremaining: 5m 31s\n",
      "265:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.05s\tremaining: 5m 31s\n",
      "266:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 9.11s\tremaining: 5m 31s\n",
      "267:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 9.13s\tremaining: 5m 31s\n",
      "268:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 9.16s\tremaining: 5m 31s\n",
      "269:\tlearn: 0.9820478\ttest: 0.4288889\tbest: 0.4366667 (216)\ttotal: 9.22s\tremaining: 5m 32s\n",
      "270:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.27s\tremaining: 5m 32s\n",
      "271:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.29s\tremaining: 5m 32s\n",
      "272:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.34s\tremaining: 5m 32s\n",
      "273:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.38s\tremaining: 5m 33s\n",
      "274:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.48s\tremaining: 5m 35s\n",
      "275:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.55s\tremaining: 5m 36s\n",
      "276:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.59s\tremaining: 5m 36s\n",
      "277:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.63s\tremaining: 5m 36s\n",
      "278:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.66s\tremaining: 5m 36s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "279:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.72s\tremaining: 5m 37s\n",
      "280:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.75s\tremaining: 5m 37s\n",
      "281:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.78s\tremaining: 5m 36s\n",
      "282:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.81s\tremaining: 5m 36s\n",
      "283:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.84s\tremaining: 5m 36s\n",
      "284:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 9.87s\tremaining: 5m 36s\n",
      "285:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.92s\tremaining: 5m 36s\n",
      "286:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 9.96s\tremaining: 5m 37s\n",
      "287:\tlearn: 0.9820478\ttest: 0.4222222\tbest: 0.4366667 (216)\ttotal: 10s\tremaining: 5m 37s\n",
      "288:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10s\tremaining: 5m 36s\n",
      "289:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10.1s\tremaining: 5m 36s\n",
      "290:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10.1s\tremaining: 5m 36s\n",
      "291:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10.1s\tremaining: 5m 35s\n",
      "292:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10.2s\tremaining: 5m 37s\n",
      "293:\tlearn: 0.9820478\ttest: 0.4155556\tbest: 0.4366667 (216)\ttotal: 10.3s\tremaining: 5m 39s\n",
      "294:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.3s\tremaining: 5m 39s\n",
      "295:\tlearn: 0.9820478\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.4s\tremaining: 5m 40s\n",
      "296:\tlearn: 0.9837398\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 10.4s\tremaining: 5m 41s\n",
      "297:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.5s\tremaining: 5m 41s\n",
      "298:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.5s\tremaining: 5m 41s\n",
      "299:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.6s\tremaining: 5m 41s\n",
      "300:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.6s\tremaining: 5m 41s\n",
      "301:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.7s\tremaining: 5m 42s\n",
      "302:\tlearn: 0.9837398\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.7s\tremaining: 5m 42s\n",
      "303:\tlearn: 0.9918699\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.7s\tremaining: 5m 42s\n",
      "304:\tlearn: 0.9918699\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.8s\tremaining: 5m 42s\n",
      "305:\tlearn: 0.9901779\ttest: 0.4100000\tbest: 0.4366667 (216)\ttotal: 10.8s\tremaining: 5m 42s\n",
      "306:\tlearn: 0.9901779\ttest: 0.4166667\tbest: 0.4366667 (216)\ttotal: 10.8s\tremaining: 5m 41s\n",
      "307:\tlearn: 1.0000000\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 10.9s\tremaining: 5m 42s\n",
      "308:\tlearn: 0.9901779\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 10.9s\tremaining: 5m 41s\n",
      "309:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 10.9s\tremaining: 5m 41s\n",
      "310:\tlearn: 0.9901779\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11s\tremaining: 5m 41s\n",
      "311:\tlearn: 0.9901779\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11s\tremaining: 5m 41s\n",
      "312:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 11s\tremaining: 5m 41s\n",
      "313:\tlearn: 1.0000000\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.1s\tremaining: 5m 41s\n",
      "314:\tlearn: 0.9983080\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 11.1s\tremaining: 5m 41s\n",
      "315:\tlearn: 0.9983080\ttest: 0.4233333\tbest: 0.4366667 (216)\ttotal: 11.1s\tremaining: 5m 41s\n",
      "316:\tlearn: 0.9983080\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.2s\tremaining: 5m 41s\n",
      "317:\tlearn: 0.9983080\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.2s\tremaining: 5m 41s\n",
      "318:\tlearn: 0.9983080\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.3s\tremaining: 5m 41s\n",
      "319:\tlearn: 0.9983080\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.3s\tremaining: 5m 42s\n",
      "320:\tlearn: 0.9983080\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.3s\tremaining: 5m 41s\n",
      "321:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 11.4s\tremaining: 5m 42s\n",
      "322:\tlearn: 1.0000000\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.4s\tremaining: 5m 42s\n",
      "323:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.4s\tremaining: 5m 41s\n",
      "324:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.5s\tremaining: 5m 41s\n",
      "325:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.5s\tremaining: 5m 41s\n",
      "326:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.6s\tremaining: 5m 41s\n",
      "327:\tlearn: 1.0000000\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.6s\tremaining: 5m 41s\n",
      "328:\tlearn: 1.0000000\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 11.6s\tremaining: 5m 41s\n",
      "329:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.7s\tremaining: 5m 41s\n",
      "330:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.7s\tremaining: 5m 41s\n",
      "331:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.7s\tremaining: 5m 41s\n",
      "332:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.8s\tremaining: 5m 41s\n",
      "333:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.8s\tremaining: 5m 41s\n",
      "334:\tlearn: 1.0000000\ttest: 0.4111111\tbest: 0.4366667 (216)\ttotal: 11.9s\tremaining: 5m 42s\n",
      "335:\tlearn: 1.0000000\ttest: 0.4055556\tbest: 0.4366667 (216)\ttotal: 11.9s\tremaining: 5m 42s\n",
      "336:\tlearn: 1.0000000\ttest: 0.4055556\tbest: 0.4366667 (216)\ttotal: 11.9s\tremaining: 5m 41s\n",
      "337:\tlearn: 1.0000000\ttest: 0.4122222\tbest: 0.4366667 (216)\ttotal: 12s\tremaining: 5m 41s\n",
      "338:\tlearn: 1.0000000\ttest: 0.4122222\tbest: 0.4366667 (216)\ttotal: 12s\tremaining: 5m 41s\n",
      "339:\tlearn: 1.0000000\ttest: 0.4177778\tbest: 0.4366667 (216)\ttotal: 12s\tremaining: 5m 41s\n",
      "340:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.1s\tremaining: 5m 41s\n",
      "341:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.1s\tremaining: 5m 41s\n",
      "342:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.1s\tremaining: 5m 41s\n",
      "343:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.1s\tremaining: 5m 40s\n",
      "344:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.2s\tremaining: 5m 40s\n",
      "345:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.2s\tremaining: 5m 40s\n",
      "346:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.2s\tremaining: 5m 40s\n",
      "347:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.3s\tremaining: 5m 40s\n",
      "348:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.3s\tremaining: 5m 39s\n",
      "349:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.3s\tremaining: 5m 39s\n",
      "350:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.4s\tremaining: 5m 39s\n",
      "351:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4366667 (216)\ttotal: 12.4s\tremaining: 5m 39s\n",
      "352:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.4s\tremaining: 5m 39s\n",
      "353:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.5s\tremaining: 5m 40s\n",
      "354:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.5s\tremaining: 5m 40s\n",
      "355:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.6s\tremaining: 5m 40s\n",
      "356:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4366667 (216)\ttotal: 12.6s\tremaining: 5m 40s\n",
      "357:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4366667 (216)\ttotal: 12.6s\tremaining: 5m 40s\n",
      "358:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4366667 (216)\ttotal: 12.7s\tremaining: 5m 40s\n",
      "359:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4366667 (216)\ttotal: 12.7s\tremaining: 5m 40s\n",
      "360:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.7s\tremaining: 5m 40s\n",
      "361:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4366667 (216)\ttotal: 12.8s\tremaining: 5m 39s\n",
      "362:\tlearn: 1.0000000\ttest: 0.4188889\tbest: 0.4366667 (216)\ttotal: 12.8s\tremaining: 5m 39s\n",
      "363:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4366667 (216)\ttotal: 12.8s\tremaining: 5m 39s\n",
      "364:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4366667 (216)\ttotal: 12.9s\tremaining: 5m 39s\n",
      "365:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 12.9s\tremaining: 5m 39s\n",
      "366:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 12.9s\tremaining: 5m 39s\n",
      "367:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13s\tremaining: 5m 39s\n",
      "368:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13s\tremaining: 5m 39s\n",
      "369:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13s\tremaining: 5m 38s\n",
      "370:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13s\tremaining: 5m 38s\n",
      "371:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13.1s\tremaining: 5m 38s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "372:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13.1s\tremaining: 5m 38s\n",
      "373:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13.1s\tremaining: 5m 38s\n",
      "374:\tlearn: 1.0000000\ttest: 0.4444444\tbest: 0.4444444 (365)\ttotal: 13.2s\tremaining: 5m 38s\n",
      "375:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 13.2s\tremaining: 5m 37s\n",
      "376:\tlearn: 1.0000000\ttest: 0.4322222\tbest: 0.4444444 (365)\ttotal: 13.2s\tremaining: 5m 37s\n",
      "377:\tlearn: 1.0000000\ttest: 0.4322222\tbest: 0.4444444 (365)\ttotal: 13.2s\tremaining: 5m 37s\n",
      "378:\tlearn: 1.0000000\ttest: 0.4322222\tbest: 0.4444444 (365)\ttotal: 13.3s\tremaining: 5m 37s\n",
      "379:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 13.3s\tremaining: 5m 36s\n",
      "380:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.3s\tremaining: 5m 36s\n",
      "381:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 13.4s\tremaining: 5m 37s\n",
      "382:\tlearn: 1.0000000\ttest: 0.4322222\tbest: 0.4444444 (365)\ttotal: 13.5s\tremaining: 5m 38s\n",
      "383:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 13.5s\tremaining: 5m 38s\n",
      "384:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.5s\tremaining: 5m 38s\n",
      "385:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.6s\tremaining: 5m 38s\n",
      "386:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.6s\tremaining: 5m 38s\n",
      "387:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.6s\tremaining: 5m 37s\n",
      "388:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.7s\tremaining: 5m 37s\n",
      "389:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.7s\tremaining: 5m 37s\n",
      "390:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.7s\tremaining: 5m 37s\n",
      "391:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.8s\tremaining: 5m 37s\n",
      "392:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 13.8s\tremaining: 5m 37s\n",
      "393:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 13.8s\tremaining: 5m 37s\n",
      "394:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 13.9s\tremaining: 5m 37s\n",
      "395:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 13.9s\tremaining: 5m 36s\n",
      "396:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 13.9s\tremaining: 5m 36s\n",
      "397:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14s\tremaining: 5m 37s\n",
      "398:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14s\tremaining: 5m 37s\n",
      "399:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14.1s\tremaining: 5m 37s\n",
      "400:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14.1s\tremaining: 5m 37s\n",
      "401:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.1s\tremaining: 5m 37s\n",
      "402:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.2s\tremaining: 5m 38s\n",
      "403:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.2s\tremaining: 5m 38s\n",
      "404:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14.3s\tremaining: 5m 37s\n",
      "405:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.3s\tremaining: 5m 37s\n",
      "406:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.3s\tremaining: 5m 37s\n",
      "407:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.4s\tremaining: 5m 37s\n",
      "408:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 14.4s\tremaining: 5m 37s\n",
      "409:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 14.4s\tremaining: 5m 37s\n",
      "410:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.5s\tremaining: 5m 37s\n",
      "411:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 14.5s\tremaining: 5m 37s\n",
      "412:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.5s\tremaining: 5m 37s\n",
      "413:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 14.6s\tremaining: 5m 37s\n",
      "414:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.6s\tremaining: 5m 37s\n",
      "415:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 14.7s\tremaining: 5m 37s\n",
      "416:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 14.7s\tremaining: 5m 37s\n",
      "417:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.7s\tremaining: 5m 37s\n",
      "418:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 14.7s\tremaining: 5m 36s\n",
      "419:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 14.8s\tremaining: 5m 37s\n",
      "420:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 14.8s\tremaining: 5m 37s\n",
      "421:\tlearn: 1.0000000\ttest: 0.4255556\tbest: 0.4444444 (365)\ttotal: 14.8s\tremaining: 5m 36s\n",
      "422:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.9s\tremaining: 5m 36s\n",
      "423:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.9s\tremaining: 5m 36s\n",
      "424:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 14.9s\tremaining: 5m 36s\n",
      "425:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 14.9s\tremaining: 5m 35s\n",
      "426:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15s\tremaining: 5m 35s\n",
      "427:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15s\tremaining: 5m 35s\n",
      "428:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15s\tremaining: 5m 35s\n",
      "429:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.1s\tremaining: 5m 35s\n",
      "430:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 15.1s\tremaining: 5m 35s\n",
      "431:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 15.1s\tremaining: 5m 35s\n",
      "432:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.2s\tremaining: 5m 35s\n",
      "433:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.2s\tremaining: 5m 35s\n",
      "434:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.2s\tremaining: 5m 34s\n",
      "435:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.3s\tremaining: 5m 35s\n",
      "436:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.3s\tremaining: 5m 34s\n",
      "437:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.3s\tremaining: 5m 34s\n",
      "438:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.4s\tremaining: 5m 34s\n",
      "439:\tlearn: 1.0000000\ttest: 0.4244444\tbest: 0.4444444 (365)\ttotal: 15.4s\tremaining: 5m 34s\n",
      "440:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 15.4s\tremaining: 5m 34s\n",
      "441:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.5s\tremaining: 5m 34s\n",
      "442:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 15.5s\tremaining: 5m 34s\n",
      "443:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.5s\tremaining: 5m 34s\n",
      "444:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.6s\tremaining: 5m 34s\n",
      "445:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.6s\tremaining: 5m 34s\n",
      "446:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 15.6s\tremaining: 5m 34s\n",
      "447:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.7s\tremaining: 5m 34s\n",
      "448:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 15.7s\tremaining: 5m 34s\n",
      "449:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 15.7s\tremaining: 5m 33s\n",
      "450:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 15.8s\tremaining: 5m 33s\n",
      "451:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 15.8s\tremaining: 5m 33s\n",
      "452:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 15.8s\tremaining: 5m 33s\n",
      "453:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 15.9s\tremaining: 5m 33s\n",
      "454:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 15.9s\tremaining: 5m 33s\n",
      "455:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 16s\tremaining: 5m 33s\n",
      "456:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16s\tremaining: 5m 33s\n",
      "457:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16s\tremaining: 5m 33s\n",
      "458:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.1s\tremaining: 5m 33s\n",
      "459:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.1s\tremaining: 5m 33s\n",
      "460:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.1s\tremaining: 5m 33s\n",
      "461:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.1s\tremaining: 5m 33s\n",
      "462:\tlearn: 1.0000000\ttest: 0.4377778\tbest: 0.4444444 (365)\ttotal: 16.2s\tremaining: 5m 32s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "463:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.2s\tremaining: 5m 33s\n",
      "464:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.2s\tremaining: 5m 33s\n",
      "465:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.3s\tremaining: 5m 32s\n",
      "466:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.3s\tremaining: 5m 32s\n",
      "467:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.3s\tremaining: 5m 32s\n",
      "468:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.3s\tremaining: 5m 32s\n",
      "469:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.4s\tremaining: 5m 32s\n",
      "470:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.4s\tremaining: 5m 31s\n",
      "471:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.4s\tremaining: 5m 31s\n",
      "472:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.5s\tremaining: 5m 31s\n",
      "473:\tlearn: 1.0000000\ttest: 0.4311111\tbest: 0.4444444 (365)\ttotal: 16.5s\tremaining: 5m 31s\n",
      "474:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.5s\tremaining: 5m 31s\n",
      "475:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.5s\tremaining: 5m 31s\n",
      "476:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.6s\tremaining: 5m 31s\n",
      "477:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.6s\tremaining: 5m 30s\n",
      "478:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.6s\tremaining: 5m 30s\n",
      "479:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 16.7s\tremaining: 5m 30s\n",
      "480:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.7s\tremaining: 5m 30s\n",
      "481:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.7s\tremaining: 5m 30s\n",
      "482:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 16.8s\tremaining: 5m 30s\n",
      "483:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 16.8s\tremaining: 5m 30s\n",
      "484:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 16.9s\tremaining: 5m 30s\n",
      "485:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.9s\tremaining: 5m 30s\n",
      "486:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 16.9s\tremaining: 5m 30s\n",
      "487:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 16.9s\tremaining: 5m 30s\n",
      "488:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17s\tremaining: 5m 30s\n",
      "489:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17s\tremaining: 5m 30s\n",
      "490:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17s\tremaining: 5m 30s\n",
      "491:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.1s\tremaining: 5m 29s\n",
      "492:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.1s\tremaining: 5m 29s\n",
      "493:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.1s\tremaining: 5m 29s\n",
      "494:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.2s\tremaining: 5m 29s\n",
      "495:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.2s\tremaining: 5m 29s\n",
      "496:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.2s\tremaining: 5m 29s\n",
      "497:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.3s\tremaining: 5m 29s\n",
      "498:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.3s\tremaining: 5m 29s\n",
      "499:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 17.3s\tremaining: 5m 29s\n",
      "500:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 17.4s\tremaining: 5m 29s\n",
      "501:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.4s\tremaining: 5m 29s\n",
      "502:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 17.4s\tremaining: 5m 29s\n",
      "503:\tlearn: 1.0000000\ttest: 0.4300000\tbest: 0.4444444 (365)\ttotal: 17.5s\tremaining: 5m 28s\n",
      "504:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.5s\tremaining: 5m 28s\n",
      "505:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.5s\tremaining: 5m 28s\n",
      "506:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.5s\tremaining: 5m 28s\n",
      "507:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.6s\tremaining: 5m 28s\n",
      "508:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.6s\tremaining: 5m 28s\n",
      "509:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.6s\tremaining: 5m 28s\n",
      "510:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.7s\tremaining: 5m 28s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200529/models/CatboostClassifier/model.pkl\n",
      "\t0.4444\t = Validation balanced_accuracy score\n",
      "\t18.07s\t = Training runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "511:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.7s\tremaining: 5m 28s\n",
      "512:\tlearn: 1.0000000\ttest: 0.4366667\tbest: 0.4444444 (365)\ttotal: 17.8s\tremaining: 5m 28s\n",
      "513:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.8s\tremaining: 5m 28s\n",
      "514:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.8s\tremaining: 5m 28s\n",
      "515:\tlearn: 1.0000000\ttest: 0.4433333\tbest: 0.4444444 (365)\ttotal: 17.9s\tremaining: 5m 28s\n",
      "Stopped by overfitting detector  (150 iterations wait)\n",
      "\n",
      "bestTest = 0.4444444444\n",
      "bestIteration = 365\n",
      "\n",
      "Shrink model to first 366 iterations.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitting model: NeuralNetClassifier ...\n",
      "AutoGluon Neural Network infers features are of the following types:\n",
      "{\n",
      "    \"continuous\": [\n",
      "        \"time_in_hospital\",\n",
      "        \"num_lab_procedures\",\n",
      "        \"num_procedures\",\n",
      "        \"number_diagnoses\"\n",
      "    ],\n",
      "    \"skewed\": [\n",
      "        \"num_medications\",\n",
      "        \"number_outpatient\",\n",
      "        \"number_inpatient\"\n",
      "    ],\n",
      "    \"onehot\": [\n",
      "        \"number_emergency\",\n",
      "        \"gender\",\n",
      "        \"metformin\",\n",
      "        \"tolbutamide\",\n",
      "        \"pioglitazone\",\n",
      "        \"rosiglitazone\",\n",
      "        \"acarbose\",\n",
      "        \"troglitazone\",\n",
      "        \"tolazamide\",\n",
      "        \"change\",\n",
      "        \"diabetesMed\"\n",
      "    ],\n",
      "    \"embed\": [\n",
      "        \"age\",\n",
      "        \"admission_type_id\",\n",
      "        \"discharge_disposition_id\",\n",
      "        \"admission_source_id\",\n",
      "        \"medical_specialty\",\n",
      "        \"diag_1\",\n",
      "        \"diag_2\",\n",
      "        \"diag_3\",\n",
      "        \"max_glu_serum\",\n",
      "        \"A1Cresult\",\n",
      "        \"repaglinide\",\n",
      "        \"glimepiride\",\n",
      "        \"glipizide\",\n",
      "        \"glyburide\",\n",
      "        \"insulin\"\n",
      "    ],\n",
      "    \"language\": []\n",
      "}\n",
      "\n",
      "\n",
      "Training data for neural network has: 480 examples, 33 features (18 vector, 15 embedding, 0 language)\n",
      "Training neural network for up to 500 epochs...\n",
      "initializing neural network...\n",
      "initialized\n",
      "Neural network architecture:\n",
      "EmbedNet(\n",
      "  (numeric_block): NumericBlock(\n",
      "    (body): Dense(None -> 225, Activation(relu))\n",
      "  )\n",
      "  (embed_blocks): HybridSequential(\n",
      "    (0): EmbedBlock(\n",
      "      (body): Embedding(11 -> 6, float32)\n",
      "    )\n",
      "    (1): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (2): EmbedBlock(\n",
      "      (body): Embedding(9 -> 5, float32)\n",
      "    )\n",
      "    (3): EmbedBlock(\n",
      "      (body): Embedding(8 -> 5, float32)\n",
      "    )\n",
      "    (4): EmbedBlock(\n",
      "      (body): Embedding(28 -> 10, float32)\n",
      "    )\n",
      "    (5): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (6): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (7): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (8): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (9): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (10): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (11): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (12): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (13): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (14): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "  )\n",
      "  (output_block): WideAndDeepBlock(\n",
      "    (deep): FeedforwardBlock(\n",
      "      (body): HybridSequential(\n",
      "        (0): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (1): Dropout(p = 0.1, axes=())\n",
      "        (2): Dense(None -> 256, Activation(relu))\n",
      "        (3): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (4): Dropout(p = 0.1, axes=())\n",
      "        (5): Dense(None -> 128, Activation(relu))\n",
      "        (6): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (7): Dropout(p = 0.1, axes=())\n",
      "        (8): Dense(None -> 3, linear)\n",
      "      )\n",
      "    )\n",
      "    (wide): Dense(None -> 3, linear)\n",
      "  )\n",
      ")\n",
      "Epoch 0.  Train loss: 1.2818825, Val balanced_accuracy: 0.24444444444444444\n",
      "Epoch 1.  Train loss: 1.2321645, Val balanced_accuracy: 0.26666666666666666\n",
      "Epoch 2.  Train loss: 1.1459966, Val balanced_accuracy: 0.26666666666666666\n",
      "Epoch 3.  Train loss: 1.1346463, Val balanced_accuracy: 0.2722222222222222\n",
      "Epoch 4.  Train loss: 1.064742, Val balanced_accuracy: 0.3\n",
      "Epoch 5.  Train loss: 1.0329398, Val balanced_accuracy: 0.3277777777777778\n",
      "Epoch 6.  Train loss: 0.9963439, Val balanced_accuracy: 0.3177777777777778\n",
      "Epoch 7.  Train loss: 0.9672677, Val balanced_accuracy: 0.34\n",
      "Epoch 8.  Train loss: 0.92278713, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 9.  Train loss: 0.87299246, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 10.  Train loss: 0.8555339, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 11.  Train loss: 0.834727, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 12.  Train loss: 0.7900263, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 13.  Train loss: 0.7556203, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 14.  Train loss: 0.74370325, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 15.  Train loss: 0.7080128, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 16.  Train loss: 0.67903405, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 17.  Train loss: 0.66799074, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 18.  Train loss: 0.6349434, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 19.  Train loss: 0.61664116, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 20.  Train loss: 0.5909943, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 21.  Train loss: 0.56924284, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 22.  Train loss: 0.5385761, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 23.  Train loss: 0.50534725, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 24.  Train loss: 0.48468634, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 25.  Train loss: 0.4801607, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 26.  Train loss: 0.44868293, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 27.  Train loss: 0.44377133, Val balanced_accuracy: 0.3333333333333334\n",
      "Epoch 28.  Train loss: 0.4222689, Val balanced_accuracy: 0.3333333333333334\n",
      "Best model found in epoch 7. Val balanced_accuracy: 0.34\n",
      "Saving AutogluonModels/ag-20200801_200529/models/NeuralNetClassifier/model.pkl\n",
      "\t0.34\t = Validation balanced_accuracy score\n",
      "\t4.98s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Fitting model: LightGBMClassifierCustom ...\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True, 'learning_rate': 0.03, 'num_leaves': 128, 'feature_fraction': 0.9, 'min_data_in_leaf': 3, 'seed_value': 0}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttrain_set's multi_logloss: 0.893412\ttrain_set's balanced_accuracy: 0.333333\tvalid_set's multi_logloss: 0.915573\tvalid_set's balanced_accuracy: 0.333333\n",
      "[2]\ttrain_set's multi_logloss: 0.866556\ttrain_set's balanced_accuracy: 0.333333\tvalid_set's multi_logloss: 0.911246\tvalid_set's balanced_accuracy: 0.333333\n",
      "[3]\ttrain_set's multi_logloss: 0.840547\ttrain_set's balanced_accuracy: 0.333333\tvalid_set's multi_logloss: 0.908139\tvalid_set's balanced_accuracy: 0.333333\n",
      "[4]\ttrain_set's multi_logloss: 0.815812\ttrain_set's balanced_accuracy: 0.439932\tvalid_set's multi_logloss: 0.90058\tvalid_set's balanced_accuracy: 0.346667\n",
      "[5]\ttrain_set's multi_logloss: 0.792856\ttrain_set's balanced_accuracy: 0.548223\tvalid_set's multi_logloss: 0.898482\tvalid_set's balanced_accuracy: 0.347778\n",
      "[6]\ttrain_set's multi_logloss: 0.770575\ttrain_set's balanced_accuracy: 0.598985\tvalid_set's multi_logloss: 0.896211\tvalid_set's balanced_accuracy: 0.355556\n",
      "[7]\ttrain_set's multi_logloss: 0.749238\ttrain_set's balanced_accuracy: 0.626058\tvalid_set's multi_logloss: 0.894497\tvalid_set's balanced_accuracy: 0.341111\n",
      "[8]\ttrain_set's multi_logloss: 0.728484\ttrain_set's balanced_accuracy: 0.64467\tvalid_set's multi_logloss: 0.890342\tvalid_set's balanced_accuracy: 0.374444\n",
      "[9]\ttrain_set's multi_logloss: 0.70969\ttrain_set's balanced_accuracy: 0.656514\tvalid_set's multi_logloss: 0.888834\tvalid_set's balanced_accuracy: 0.382222\n",
      "[10]\ttrain_set's multi_logloss: 0.690392\ttrain_set's balanced_accuracy: 0.656514\tvalid_set's multi_logloss: 0.884859\tvalid_set's balanced_accuracy: 0.378889\n",
      "[11]\ttrain_set's multi_logloss: 0.670949\ttrain_set's balanced_accuracy: 0.656514\tvalid_set's multi_logloss: 0.883968\tvalid_set's balanced_accuracy: 0.385556\n",
      "[12]\ttrain_set's multi_logloss: 0.653788\ttrain_set's balanced_accuracy: 0.658206\tvalid_set's multi_logloss: 0.882343\tvalid_set's balanced_accuracy: 0.392222\n",
      "[13]\ttrain_set's multi_logloss: 0.636231\ttrain_set's balanced_accuracy: 0.659898\tvalid_set's multi_logloss: 0.882013\tvalid_set's balanced_accuracy: 0.401111\n",
      "[14]\ttrain_set's multi_logloss: 0.619831\ttrain_set's balanced_accuracy: 0.659898\tvalid_set's multi_logloss: 0.880486\tvalid_set's balanced_accuracy: 0.395556\n",
      "[15]\ttrain_set's multi_logloss: 0.603711\ttrain_set's balanced_accuracy: 0.663283\tvalid_set's multi_logloss: 0.881148\tvalid_set's balanced_accuracy: 0.395556\n",
      "[16]\ttrain_set's multi_logloss: 0.588731\ttrain_set's balanced_accuracy: 0.664975\tvalid_set's multi_logloss: 0.878595\tvalid_set's balanced_accuracy: 0.402222\n",
      "[17]\ttrain_set's multi_logloss: 0.57346\ttrain_set's balanced_accuracy: 0.664975\tvalid_set's multi_logloss: 0.876992\tvalid_set's balanced_accuracy: 0.395556\n",
      "[18]\ttrain_set's multi_logloss: 0.559212\ttrain_set's balanced_accuracy: 0.664975\tvalid_set's multi_logloss: 0.874071\tvalid_set's balanced_accuracy: 0.395556\n",
      "[19]\ttrain_set's multi_logloss: 0.544434\ttrain_set's balanced_accuracy: 0.666667\tvalid_set's multi_logloss: 0.872788\tvalid_set's balanced_accuracy: 0.408889\n",
      "[20]\ttrain_set's multi_logloss: 0.53101\ttrain_set's balanced_accuracy: 0.682927\tvalid_set's multi_logloss: 0.872521\tvalid_set's balanced_accuracy: 0.428889\n",
      "[21]\ttrain_set's multi_logloss: 0.517956\ttrain_set's balanced_accuracy: 0.682927\tvalid_set's multi_logloss: 0.872359\tvalid_set's balanced_accuracy: 0.43\n",
      "[22]\ttrain_set's multi_logloss: 0.505583\ttrain_set's balanced_accuracy: 0.723577\tvalid_set's multi_logloss: 0.86834\tvalid_set's balanced_accuracy: 0.43\n",
      "[23]\ttrain_set's multi_logloss: 0.493724\ttrain_set's balanced_accuracy: 0.772358\tvalid_set's multi_logloss: 0.869016\tvalid_set's balanced_accuracy: 0.411111\n",
      "[24]\ttrain_set's multi_logloss: 0.481924\ttrain_set's balanced_accuracy: 0.788618\tvalid_set's multi_logloss: 0.868769\tvalid_set's balanced_accuracy: 0.423333\n",
      "[25]\ttrain_set's multi_logloss: 0.470006\ttrain_set's balanced_accuracy: 0.813008\tvalid_set's multi_logloss: 0.868523\tvalid_set's balanced_accuracy: 0.405556\n",
      "[26]\ttrain_set's multi_logloss: 0.458816\ttrain_set's balanced_accuracy: 0.829268\tvalid_set's multi_logloss: 0.868053\tvalid_set's balanced_accuracy: 0.411111\n",
      "[27]\ttrain_set's multi_logloss: 0.447452\ttrain_set's balanced_accuracy: 0.853659\tvalid_set's multi_logloss: 0.866778\tvalid_set's balanced_accuracy: 0.411111\n",
      "[28]\ttrain_set's multi_logloss: 0.436719\ttrain_set's balanced_accuracy: 0.910569\tvalid_set's multi_logloss: 0.867193\tvalid_set's balanced_accuracy: 0.417778\n",
      "[29]\ttrain_set's multi_logloss: 0.426524\ttrain_set's balanced_accuracy: 0.926829\tvalid_set's multi_logloss: 0.865851\tvalid_set's balanced_accuracy: 0.417778\n",
      "[30]\ttrain_set's multi_logloss: 0.415846\ttrain_set's balanced_accuracy: 0.943089\tvalid_set's multi_logloss: 0.866186\tvalid_set's balanced_accuracy: 0.417778\n",
      "[31]\ttrain_set's multi_logloss: 0.405724\ttrain_set's balanced_accuracy: 0.96748\tvalid_set's multi_logloss: 0.866448\tvalid_set's balanced_accuracy: 0.417778\n",
      "[32]\ttrain_set's multi_logloss: 0.396723\ttrain_set's balanced_accuracy: 0.98374\tvalid_set's multi_logloss: 0.865207\tvalid_set's balanced_accuracy: 0.417778\n",
      "[33]\ttrain_set's multi_logloss: 0.38702\ttrain_set's balanced_accuracy: 0.99187\tvalid_set's multi_logloss: 0.864064\tvalid_set's balanced_accuracy: 0.406667\n",
      "[34]\ttrain_set's multi_logloss: 0.377572\ttrain_set's balanced_accuracy: 0.99187\tvalid_set's multi_logloss: 0.863889\tvalid_set's balanced_accuracy: 0.406667\n",
      "[35]\ttrain_set's multi_logloss: 0.368811\ttrain_set's balanced_accuracy: 0.99187\tvalid_set's multi_logloss: 0.863951\tvalid_set's balanced_accuracy: 0.406667\n",
      "[36]\ttrain_set's multi_logloss: 0.360083\ttrain_set's balanced_accuracy: 0.99187\tvalid_set's multi_logloss: 0.864504\tvalid_set's balanced_accuracy: 0.406667\n",
      "[37]\ttrain_set's multi_logloss: 0.351814\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.864231\tvalid_set's balanced_accuracy: 0.406667\n",
      "[38]\ttrain_set's multi_logloss: 0.343458\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.864965\tvalid_set's balanced_accuracy: 0.406667\n",
      "[39]\ttrain_set's multi_logloss: 0.335472\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.866007\tvalid_set's balanced_accuracy: 0.406667\n",
      "[40]\ttrain_set's multi_logloss: 0.327457\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.865104\tvalid_set's balanced_accuracy: 0.406667\n",
      "[41]\ttrain_set's multi_logloss: 0.319415\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.864233\tvalid_set's balanced_accuracy: 0.413333\n",
      "[42]\ttrain_set's multi_logloss: 0.311837\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.864229\tvalid_set's balanced_accuracy: 0.446667\n",
      "[43]\ttrain_set's multi_logloss: 0.30461\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.863263\tvalid_set's balanced_accuracy: 0.446667\n",
      "[44]\ttrain_set's multi_logloss: 0.297748\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.863802\tvalid_set's balanced_accuracy: 0.446667\n",
      "[45]\ttrain_set's multi_logloss: 0.290868\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.865633\tvalid_set's balanced_accuracy: 0.441111\n",
      "[46]\ttrain_set's multi_logloss: 0.283952\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.866434\tvalid_set's balanced_accuracy: 0.44\n",
      "[47]\ttrain_set's multi_logloss: 0.277287\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.867163\tvalid_set's balanced_accuracy: 0.446667\n",
      "[48]\ttrain_set's multi_logloss: 0.270877\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.868096\tvalid_set's balanced_accuracy: 0.44\n",
      "[49]\ttrain_set's multi_logloss: 0.264813\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.868108\tvalid_set's balanced_accuracy: 0.412222\n",
      "[50]\ttrain_set's multi_logloss: 0.258502\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.869975\tvalid_set's balanced_accuracy: 0.412222\n",
      "[51]\ttrain_set's multi_logloss: 0.253005\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.870342\tvalid_set's balanced_accuracy: 0.412222\n",
      "[52]\ttrain_set's multi_logloss: 0.247142\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.869795\tvalid_set's balanced_accuracy: 0.412222\n",
      "[53]\ttrain_set's multi_logloss: 0.24138\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.872056\tvalid_set's balanced_accuracy: 0.405556\n",
      "[54]\ttrain_set's multi_logloss: 0.235957\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.87148\tvalid_set's balanced_accuracy: 0.411111\n",
      "[55]\ttrain_set's multi_logloss: 0.230582\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.872832\tvalid_set's balanced_accuracy: 0.405556\n",
      "[56]\ttrain_set's multi_logloss: 0.22518\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.873804\tvalid_set's balanced_accuracy: 0.405556\n",
      "[57]\ttrain_set's multi_logloss: 0.219993\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.874833\tvalid_set's balanced_accuracy: 0.405556\n",
      "[58]\ttrain_set's multi_logloss: 0.215322\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.875736\tvalid_set's balanced_accuracy: 0.405556\n",
      "[59]\ttrain_set's multi_logloss: 0.210646\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.876689\tvalid_set's balanced_accuracy: 0.405556\n",
      "[60]\ttrain_set's multi_logloss: 0.205939\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.876279\tvalid_set's balanced_accuracy: 0.405556\n",
      "[61]\ttrain_set's multi_logloss: 0.201485\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.875871\tvalid_set's balanced_accuracy: 0.405556\n",
      "[62]\ttrain_set's multi_logloss: 0.19693\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.875455\tvalid_set's balanced_accuracy: 0.411111\n",
      "[63]\ttrain_set's multi_logloss: 0.192662\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.875541\tvalid_set's balanced_accuracy: 0.405556\n",
      "[64]\ttrain_set's multi_logloss: 0.188368\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.876386\tvalid_set's balanced_accuracy: 0.405556\n",
      "[65]\ttrain_set's multi_logloss: 0.184131\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.877942\tvalid_set's balanced_accuracy: 0.405556\n",
      "[66]\ttrain_set's multi_logloss: 0.180104\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.878813\tvalid_set's balanced_accuracy: 0.405556\n",
      "[67]\ttrain_set's multi_logloss: 0.17595\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.879867\tvalid_set's balanced_accuracy: 0.412222\n",
      "[68]\ttrain_set's multi_logloss: 0.17204\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.880648\tvalid_set's balanced_accuracy: 0.405556\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[69]\ttrain_set's multi_logloss: 0.168251\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.882719\tvalid_set's balanced_accuracy: 0.405556\n",
      "[70]\ttrain_set's multi_logloss: 0.164613\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.883237\tvalid_set's balanced_accuracy: 0.412222\n",
      "[71]\ttrain_set's multi_logloss: 0.160898\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.884243\tvalid_set's balanced_accuracy: 0.406667\n",
      "[72]\ttrain_set's multi_logloss: 0.157341\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.885666\tvalid_set's balanced_accuracy: 0.395556\n",
      "[73]\ttrain_set's multi_logloss: 0.153809\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.886188\tvalid_set's balanced_accuracy: 0.395556\n",
      "[74]\ttrain_set's multi_logloss: 0.150394\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.887558\tvalid_set's balanced_accuracy: 0.401111\n",
      "[75]\ttrain_set's multi_logloss: 0.147107\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.888935\tvalid_set's balanced_accuracy: 0.401111\n",
      "[76]\ttrain_set's multi_logloss: 0.143914\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.888386\tvalid_set's balanced_accuracy: 0.401111\n",
      "[77]\ttrain_set's multi_logloss: 0.140805\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.891536\tvalid_set's balanced_accuracy: 0.401111\n",
      "[78]\ttrain_set's multi_logloss: 0.137588\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.893176\tvalid_set's balanced_accuracy: 0.401111\n",
      "[79]\ttrain_set's multi_logloss: 0.134494\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.893367\tvalid_set's balanced_accuracy: 0.401111\n",
      "[80]\ttrain_set's multi_logloss: 0.131744\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.894254\tvalid_set's balanced_accuracy: 0.401111\n",
      "[81]\ttrain_set's multi_logloss: 0.128955\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.89621\tvalid_set's balanced_accuracy: 0.401111\n",
      "[82]\ttrain_set's multi_logloss: 0.12619\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.897523\tvalid_set's balanced_accuracy: 0.394444\n",
      "[83]\ttrain_set's multi_logloss: 0.123353\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.899897\tvalid_set's balanced_accuracy: 0.394444\n",
      "[84]\ttrain_set's multi_logloss: 0.120701\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.899868\tvalid_set's balanced_accuracy: 0.394444\n",
      "[85]\ttrain_set's multi_logloss: 0.11803\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.901911\tvalid_set's balanced_accuracy: 0.394444\n",
      "[86]\ttrain_set's multi_logloss: 0.115578\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.905468\tvalid_set's balanced_accuracy: 0.394444\n",
      "[87]\ttrain_set's multi_logloss: 0.113057\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.907982\tvalid_set's balanced_accuracy: 0.394444\n",
      "[88]\ttrain_set's multi_logloss: 0.11061\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.911547\tvalid_set's balanced_accuracy: 0.394444\n",
      "[89]\ttrain_set's multi_logloss: 0.108111\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.912068\tvalid_set's balanced_accuracy: 0.394444\n",
      "[90]\ttrain_set's multi_logloss: 0.105835\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.912336\tvalid_set's balanced_accuracy: 0.4\n",
      "[91]\ttrain_set's multi_logloss: 0.103437\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.914898\tvalid_set's balanced_accuracy: 0.394444\n",
      "[92]\ttrain_set's multi_logloss: 0.101235\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.91488\tvalid_set's balanced_accuracy: 0.4\n",
      "[93]\ttrain_set's multi_logloss: 0.0990319\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.916183\tvalid_set's balanced_accuracy: 0.4\n",
      "[94]\ttrain_set's multi_logloss: 0.0969542\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.917199\tvalid_set's balanced_accuracy: 0.405556\n",
      "[95]\ttrain_set's multi_logloss: 0.0947827\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.917568\tvalid_set's balanced_accuracy: 0.411111\n",
      "[96]\ttrain_set's multi_logloss: 0.0929508\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.919246\tvalid_set's balanced_accuracy: 0.411111\n",
      "[97]\ttrain_set's multi_logloss: 0.0909559\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.919462\tvalid_set's balanced_accuracy: 0.405556\n",
      "[98]\ttrain_set's multi_logloss: 0.0889423\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.918404\tvalid_set's balanced_accuracy: 0.411111\n",
      "[99]\ttrain_set's multi_logloss: 0.0870354\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.917284\tvalid_set's balanced_accuracy: 0.411111\n",
      "[100]\ttrain_set's multi_logloss: 0.0850915\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.91811\tvalid_set's balanced_accuracy: 0.405556\n",
      "[101]\ttrain_set's multi_logloss: 0.083285\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.919117\tvalid_set's balanced_accuracy: 0.4\n",
      "[102]\ttrain_set's multi_logloss: 0.081509\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.918953\tvalid_set's balanced_accuracy: 0.4\n",
      "[103]\ttrain_set's multi_logloss: 0.0797808\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.919664\tvalid_set's balanced_accuracy: 0.4\n",
      "[104]\ttrain_set's multi_logloss: 0.0780487\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.919834\tvalid_set's balanced_accuracy: 0.406667\n",
      "[105]\ttrain_set's multi_logloss: 0.0764513\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.921074\tvalid_set's balanced_accuracy: 0.406667\n",
      "[106]\ttrain_set's multi_logloss: 0.0748006\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.921428\tvalid_set's balanced_accuracy: 0.406667\n",
      "[107]\ttrain_set's multi_logloss: 0.0732021\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.923606\tvalid_set's balanced_accuracy: 0.406667\n",
      "[108]\ttrain_set's multi_logloss: 0.0716233\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.923287\tvalid_set's balanced_accuracy: 0.406667\n",
      "[109]\ttrain_set's multi_logloss: 0.0700765\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.923635\tvalid_set's balanced_accuracy: 0.406667\n",
      "[110]\ttrain_set's multi_logloss: 0.0685436\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.9241\tvalid_set's balanced_accuracy: 0.401111\n",
      "[111]\ttrain_set's multi_logloss: 0.0670458\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.924284\tvalid_set's balanced_accuracy: 0.401111\n",
      "[112]\ttrain_set's multi_logloss: 0.0656227\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.92387\tvalid_set's balanced_accuracy: 0.4\n",
      "[113]\ttrain_set's multi_logloss: 0.0642373\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.925677\tvalid_set's balanced_accuracy: 0.4\n",
      "[114]\ttrain_set's multi_logloss: 0.062895\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.926307\tvalid_set's balanced_accuracy: 0.4\n",
      "[115]\ttrain_set's multi_logloss: 0.0615773\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.926446\tvalid_set's balanced_accuracy: 0.405556\n",
      "[116]\ttrain_set's multi_logloss: 0.060275\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.926739\tvalid_set's balanced_accuracy: 0.405556\n",
      "[117]\ttrain_set's multi_logloss: 0.0590171\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.926317\tvalid_set's balanced_accuracy: 0.405556\n",
      "[118]\ttrain_set's multi_logloss: 0.057713\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.927625\tvalid_set's balanced_accuracy: 0.405556\n",
      "[119]\ttrain_set's multi_logloss: 0.0564184\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.929176\tvalid_set's balanced_accuracy: 0.4\n",
      "[120]\ttrain_set's multi_logloss: 0.055201\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.929514\tvalid_set's balanced_accuracy: 0.4\n",
      "[121]\ttrain_set's multi_logloss: 0.0540324\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.930999\tvalid_set's balanced_accuracy: 0.4\n",
      "[122]\ttrain_set's multi_logloss: 0.0528742\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.931896\tvalid_set's balanced_accuracy: 0.4\n",
      "[123]\ttrain_set's multi_logloss: 0.051695\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.932788\tvalid_set's balanced_accuracy: 0.405556\n",
      "[124]\ttrain_set's multi_logloss: 0.0505971\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.934914\tvalid_set's balanced_accuracy: 0.394444\n",
      "[125]\ttrain_set's multi_logloss: 0.049571\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.935561\tvalid_set's balanced_accuracy: 0.4\n",
      "[126]\ttrain_set's multi_logloss: 0.0485686\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.937302\tvalid_set's balanced_accuracy: 0.4\n",
      "[127]\ttrain_set's multi_logloss: 0.047567\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.938401\tvalid_set's balanced_accuracy: 0.394444\n",
      "[128]\ttrain_set's multi_logloss: 0.0465587\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.939064\tvalid_set's balanced_accuracy: 0.394444\n",
      "[129]\ttrain_set's multi_logloss: 0.0456218\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.940577\tvalid_set's balanced_accuracy: 0.394444\n",
      "[130]\ttrain_set's multi_logloss: 0.0446659\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.941284\tvalid_set's balanced_accuracy: 0.394444\n",
      "[131]\ttrain_set's multi_logloss: 0.043718\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.942528\tvalid_set's balanced_accuracy: 0.401111\n",
      "[132]\ttrain_set's multi_logloss: 0.0428206\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.944278\tvalid_set's balanced_accuracy: 0.401111\n",
      "[133]\ttrain_set's multi_logloss: 0.0419152\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.945185\tvalid_set's balanced_accuracy: 0.406667\n",
      "[134]\ttrain_set's multi_logloss: 0.0410145\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.946533\tvalid_set's balanced_accuracy: 0.406667\n",
      "[135]\ttrain_set's multi_logloss: 0.0401459\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.947255\tvalid_set's balanced_accuracy: 0.406667\n",
      "[136]\ttrain_set's multi_logloss: 0.0393078\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.947553\tvalid_set's balanced_accuracy: 0.406667\n",
      "[137]\ttrain_set's multi_logloss: 0.038473\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.950226\tvalid_set's balanced_accuracy: 0.412222\n",
      "[138]\ttrain_set's multi_logloss: 0.0376611\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.951806\tvalid_set's balanced_accuracy: 0.412222\n",
      "[139]\ttrain_set's multi_logloss: 0.0368512\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.953873\tvalid_set's balanced_accuracy: 0.412222\n",
      "[140]\ttrain_set's multi_logloss: 0.0360651\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.955617\tvalid_set's balanced_accuracy: 0.412222\n",
      "[141]\ttrain_set's multi_logloss: 0.0353229\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.957128\tvalid_set's balanced_accuracy: 0.401111\n",
      "[142]\ttrain_set's multi_logloss: 0.0345654\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.957323\tvalid_set's balanced_accuracy: 0.401111\n",
      "[143]\ttrain_set's multi_logloss: 0.0338471\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.95973\tvalid_set's balanced_accuracy: 0.401111\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[144]\ttrain_set's multi_logloss: 0.033143\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.961334\tvalid_set's balanced_accuracy: 0.401111\n",
      "[145]\ttrain_set's multi_logloss: 0.0324355\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.963896\tvalid_set's balanced_accuracy: 0.401111\n",
      "[146]\ttrain_set's multi_logloss: 0.0317347\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.966543\tvalid_set's balanced_accuracy: 0.401111\n",
      "[147]\ttrain_set's multi_logloss: 0.0310618\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.968376\tvalid_set's balanced_accuracy: 0.394444\n",
      "[148]\ttrain_set's multi_logloss: 0.0304463\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.971089\tvalid_set's balanced_accuracy: 0.394444\n",
      "[149]\ttrain_set's multi_logloss: 0.0298484\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.973238\tvalid_set's balanced_accuracy: 0.394444\n",
      "[150]\ttrain_set's multi_logloss: 0.0292281\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.975388\tvalid_set's balanced_accuracy: 0.394444\n",
      "[151]\ttrain_set's multi_logloss: 0.028637\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.9766\tvalid_set's balanced_accuracy: 0.394444\n",
      "[152]\ttrain_set's multi_logloss: 0.0280411\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.978896\tvalid_set's balanced_accuracy: 0.394444\n",
      "[153]\ttrain_set's multi_logloss: 0.0274604\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.981171\tvalid_set's balanced_accuracy: 0.394444\n",
      "[154]\ttrain_set's multi_logloss: 0.0268773\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.983079\tvalid_set's balanced_accuracy: 0.394444\n",
      "[155]\ttrain_set's multi_logloss: 0.0263096\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.984619\tvalid_set's balanced_accuracy: 0.394444\n",
      "[156]\ttrain_set's multi_logloss: 0.0257552\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.985913\tvalid_set's balanced_accuracy: 0.4\n",
      "[157]\ttrain_set's multi_logloss: 0.0252351\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.988847\tvalid_set's balanced_accuracy: 0.4\n",
      "[158]\ttrain_set's multi_logloss: 0.024709\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.991713\tvalid_set's balanced_accuracy: 0.4\n",
      "[159]\ttrain_set's multi_logloss: 0.0241867\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.995213\tvalid_set's balanced_accuracy: 0.393333\n",
      "[160]\ttrain_set's multi_logloss: 0.0236956\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.997443\tvalid_set's balanced_accuracy: 0.4\n",
      "[161]\ttrain_set's multi_logloss: 0.023201\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 0.999558\tvalid_set's balanced_accuracy: 0.394444\n",
      "[162]\ttrain_set's multi_logloss: 0.0227347\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0016\tvalid_set's balanced_accuracy: 0.394444\n",
      "[163]\ttrain_set's multi_logloss: 0.0222768\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00327\tvalid_set's balanced_accuracy: 0.394444\n",
      "[164]\ttrain_set's multi_logloss: 0.021806\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00571\tvalid_set's balanced_accuracy: 0.394444\n",
      "[165]\ttrain_set's multi_logloss: 0.0213573\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.00846\tvalid_set's balanced_accuracy: 0.394444\n",
      "[166]\ttrain_set's multi_logloss: 0.0209457\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0102\tvalid_set's balanced_accuracy: 0.382222\n",
      "[167]\ttrain_set's multi_logloss: 0.0205109\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0124\tvalid_set's balanced_accuracy: 0.388889\n",
      "[168]\ttrain_set's multi_logloss: 0.0200755\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01454\tvalid_set's balanced_accuracy: 0.388889\n",
      "[169]\ttrain_set's multi_logloss: 0.0196483\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01676\tvalid_set's balanced_accuracy: 0.382222\n",
      "[170]\ttrain_set's multi_logloss: 0.019239\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.01936\tvalid_set's balanced_accuracy: 0.382222\n",
      "[171]\ttrain_set's multi_logloss: 0.0188436\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0226\tvalid_set's balanced_accuracy: 0.382222\n",
      "[172]\ttrain_set's multi_logloss: 0.0184616\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02435\tvalid_set's balanced_accuracy: 0.382222\n",
      "[173]\ttrain_set's multi_logloss: 0.0180717\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02667\tvalid_set's balanced_accuracy: 0.382222\n",
      "[174]\ttrain_set's multi_logloss: 0.0176963\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.02922\tvalid_set's balanced_accuracy: 0.382222\n",
      "[175]\ttrain_set's multi_logloss: 0.0173335\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03067\tvalid_set's balanced_accuracy: 0.382222\n",
      "[176]\ttrain_set's multi_logloss: 0.0169699\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03218\tvalid_set's balanced_accuracy: 0.382222\n",
      "[177]\ttrain_set's multi_logloss: 0.0166189\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03546\tvalid_set's balanced_accuracy: 0.382222\n",
      "[178]\ttrain_set's multi_logloss: 0.0162948\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03699\tvalid_set's balanced_accuracy: 0.382222\n",
      "[179]\ttrain_set's multi_logloss: 0.0159624\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03832\tvalid_set's balanced_accuracy: 0.382222\n",
      "[180]\ttrain_set's multi_logloss: 0.0156408\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.03992\tvalid_set's balanced_accuracy: 0.382222\n",
      "[181]\ttrain_set's multi_logloss: 0.0153155\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04215\tvalid_set's balanced_accuracy: 0.382222\n",
      "[182]\ttrain_set's multi_logloss: 0.014994\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04559\tvalid_set's balanced_accuracy: 0.382222\n",
      "[183]\ttrain_set's multi_logloss: 0.0146793\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04859\tvalid_set's balanced_accuracy: 0.382222\n",
      "[184]\ttrain_set's multi_logloss: 0.0143662\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.04915\tvalid_set's balanced_accuracy: 0.382222\n",
      "[185]\ttrain_set's multi_logloss: 0.0140664\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05245\tvalid_set's balanced_accuracy: 0.382222\n",
      "[186]\ttrain_set's multi_logloss: 0.0137692\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05586\tvalid_set's balanced_accuracy: 0.382222\n",
      "[187]\ttrain_set's multi_logloss: 0.0134891\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.05903\tvalid_set's balanced_accuracy: 0.382222\n",
      "[188]\ttrain_set's multi_logloss: 0.0132066\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06155\tvalid_set's balanced_accuracy: 0.382222\n",
      "[189]\ttrain_set's multi_logloss: 0.0129311\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06469\tvalid_set's balanced_accuracy: 0.382222\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200529/models/LightGBMClassifierCustom/model.pkl\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[190]\ttrain_set's multi_logloss: 0.0126536\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.06784\tvalid_set's balanced_accuracy: 0.382222\n",
      "[191]\ttrain_set's multi_logloss: 0.0123947\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.0708\tvalid_set's balanced_accuracy: 0.382222\n",
      "[192]\ttrain_set's multi_logloss: 0.0121585\ttrain_set's balanced_accuracy: 1\tvalid_set's multi_logloss: 1.07271\tvalid_set's balanced_accuracy: 0.382222\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.4467\t = Validation balanced_accuracy score\n",
      "\t2.04s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/RandomForestClassifierGini/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/RandomForestClassifierEntr/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierGini/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierEntr/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierUnif/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierDist/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/LightGBMClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/CatboostClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/NeuralNetClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/LightGBMClassifierCustom/model.pkl\n",
      "Fitting model: weighted_ensemble_k0_l1 ...\n",
      "Saving AutogluonModels/ag-20200801_200529/models/weighted_ensemble_k0_l1/utils/model_template.pkl\n",
      "Ensemble size: 7\n",
      "Ensemble indices: [0, 0, 0, 0, 0, 0, 1]\n",
      "Ensemble weights: \n",
      "[0.85714286 0.14285714 0.         0.         0.         0.\n",
      " 0.         0.         0.         0.        ]\n",
      "Saving AutogluonModels/ag-20200801_200529/models/weighted_ensemble_k0_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/models/weighted_ensemble_k0_l1/model.pkl\n",
      "\t0.5222\t = Validation balanced_accuracy score\n",
      "\t0.69s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "Saving AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "AutoGluon training complete, total runtime = 32.2s ...\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/trainer.pkl\n",
      "File delimiter for https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv inferred as ',' (comma). If this is incorrect, please manually load the data as a pandas DataFrame.\n",
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv | Columns = 47 / 47 | Rows = 20354 -> 20354\n",
      "Total time taken in transform: 0.05028986930847168\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/CatboostClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierEntr/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/ExtraTreesClassifierGini/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierDist/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/KNeighborsClassifierUnif/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/LightGBMClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/LightGBMClassifierCustom/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/NeuralNetClassifier/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/RandomForestClassifierEntr/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/RandomForestClassifierGini/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200529/models/weighted_ensemble_k0_l1/model.pkl\n",
      "Model scores:\n",
      "{'CatboostClassifier': 0.36574513526340774, 'ExtraTreesClassifierEntr': 0.3935651004587881, 'ExtraTreesClassifierGini': 0.40210937088013504, 'KNeighborsClassifierDist': 0.32890233612824976, 'KNeighborsClassifierUnif': 0.32383852765912563, 'LightGBMClassifier': 0.3760441385856668, 'LightGBMClassifierCustom': 0.3402533881769762, 'NeuralNetClassifier': 0.34905473817433946, 'RandomForestClassifierEntr': 0.3534830986658229, 'RandomForestClassifierGini': 0.37371460211991775, 'weighted_ensemble_k0_l1': 0.371436481569372}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balanced-Accuracy of each model on test data:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ExtraTreesClassifierGini</td>\n",
       "      <td>0.402109</td>\n",
       "      <td>0.441111</td>\n",
       "      <td>0.158669</td>\n",
       "      <td>0.126700</td>\n",
       "      <td>0.547868</td>\n",
       "      <td>0.158669</td>\n",
       "      <td>0.126700</td>\n",
       "      <td>0.547868</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ExtraTreesClassifierEntr</td>\n",
       "      <td>0.393565</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.152231</td>\n",
       "      <td>0.136706</td>\n",
       "      <td>0.543493</td>\n",
       "      <td>0.152231</td>\n",
       "      <td>0.136706</td>\n",
       "      <td>0.543493</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>LightGBMClassifier</td>\n",
       "      <td>0.376044</td>\n",
       "      <td>0.520000</td>\n",
       "      <td>0.045882</td>\n",
       "      <td>0.027615</td>\n",
       "      <td>0.840943</td>\n",
       "      <td>0.045882</td>\n",
       "      <td>0.027615</td>\n",
       "      <td>0.840943</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierGini</td>\n",
       "      <td>0.373715</td>\n",
       "      <td>0.431111</td>\n",
       "      <td>0.165529</td>\n",
       "      <td>0.130083</td>\n",
       "      <td>0.698699</td>\n",
       "      <td>0.165529</td>\n",
       "      <td>0.130083</td>\n",
       "      <td>0.698699</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.371436</td>\n",
       "      <td>0.522222</td>\n",
       "      <td>0.204413</td>\n",
       "      <td>0.165672</td>\n",
       "      <td>2.073096</td>\n",
       "      <td>0.006300</td>\n",
       "      <td>0.001351</td>\n",
       "      <td>0.688660</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CatboostClassifier</td>\n",
       "      <td>0.365745</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>0.031150</td>\n",
       "      <td>0.029343</td>\n",
       "      <td>18.074721</td>\n",
       "      <td>0.031150</td>\n",
       "      <td>0.029343</td>\n",
       "      <td>18.074721</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RandomForestClassifierEntr</td>\n",
       "      <td>0.353483</td>\n",
       "      <td>0.442222</td>\n",
       "      <td>0.146133</td>\n",
       "      <td>0.130906</td>\n",
       "      <td>0.645182</td>\n",
       "      <td>0.146133</td>\n",
       "      <td>0.130906</td>\n",
       "      <td>0.645182</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NeuralNetClassifier</td>\n",
       "      <td>0.349055</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>0.191691</td>\n",
       "      <td>0.050880</td>\n",
       "      <td>4.979828</td>\n",
       "      <td>0.191691</td>\n",
       "      <td>0.050880</td>\n",
       "      <td>4.979828</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>LightGBMClassifierCustom</td>\n",
       "      <td>0.340253</td>\n",
       "      <td>0.446667</td>\n",
       "      <td>0.044743</td>\n",
       "      <td>0.041768</td>\n",
       "      <td>2.043933</td>\n",
       "      <td>0.044743</td>\n",
       "      <td>0.041768</td>\n",
       "      <td>2.043933</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>KNeighborsClassifierDist</td>\n",
       "      <td>0.328902</td>\n",
       "      <td>0.410000</td>\n",
       "      <td>0.111893</td>\n",
       "      <td>0.109032</td>\n",
       "      <td>0.005723</td>\n",
       "      <td>0.111893</td>\n",
       "      <td>0.109032</td>\n",
       "      <td>0.005723</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>KNeighborsClassifierUnif</td>\n",
       "      <td>0.323839</td>\n",
       "      <td>0.387778</td>\n",
       "      <td>0.109776</td>\n",
       "      <td>0.113860</td>\n",
       "      <td>0.006618</td>\n",
       "      <td>0.109776</td>\n",
       "      <td>0.113860</td>\n",
       "      <td>0.006618</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         model  score_test  score_val  pred_time_test  \\\n",
       "0     ExtraTreesClassifierGini    0.402109   0.441111        0.158669   \n",
       "1     ExtraTreesClassifierEntr    0.393565   0.460000        0.152231   \n",
       "2           LightGBMClassifier    0.376044   0.520000        0.045882   \n",
       "3   RandomForestClassifierGini    0.373715   0.431111        0.165529   \n",
       "4      weighted_ensemble_k0_l1    0.371436   0.522222        0.204413   \n",
       "5           CatboostClassifier    0.365745   0.444444        0.031150   \n",
       "6   RandomForestClassifierEntr    0.353483   0.442222        0.146133   \n",
       "7          NeuralNetClassifier    0.349055   0.340000        0.191691   \n",
       "8     LightGBMClassifierCustom    0.340253   0.446667        0.044743   \n",
       "9     KNeighborsClassifierDist    0.328902   0.410000        0.111893   \n",
       "10    KNeighborsClassifierUnif    0.323839   0.387778        0.109776   \n",
       "\n",
       "    pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  \\\n",
       "0        0.126700   0.547868                 0.158669                0.126700   \n",
       "1        0.136706   0.543493                 0.152231                0.136706   \n",
       "2        0.027615   0.840943                 0.045882                0.027615   \n",
       "3        0.130083   0.698699                 0.165529                0.130083   \n",
       "4        0.165672   2.073096                 0.006300                0.001351   \n",
       "5        0.029343  18.074721                 0.031150                0.029343   \n",
       "6        0.130906   0.645182                 0.146133                0.130906   \n",
       "7        0.050880   4.979828                 0.191691                0.050880   \n",
       "8        0.041768   2.043933                 0.044743                0.041768   \n",
       "9        0.109032   0.005723                 0.111893                0.109032   \n",
       "10       0.113860   0.006618                 0.109776                0.113860   \n",
       "\n",
       "    fit_time_marginal  stack_level  can_infer  \n",
       "0            0.547868            0       True  \n",
       "1            0.543493            0       True  \n",
       "2            0.840943            0       True  \n",
       "3            0.698699            0       True  \n",
       "4            0.688660            1       True  \n",
       "5           18.074721            0       True  \n",
       "6            0.645182            0       True  \n",
       "7            4.979828            0       True  \n",
       "8            2.043933            0       True  \n",
       "9            0.005723            0       True  \n",
       "10           0.006618            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from autogluon import TabularPrediction as task\n",
    "from IPython.display import display\n",
    "\n",
    "subsample_size = 600 # experiment with larger values to try AutoGluon with larger datasets \n",
    "\n",
    "train_data_full = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/train.csv')\n",
    "train_data = train_data_full.head(subsample_size) # subsample data for faster demo\n",
    "label_column = 'readmitted'\n",
    "predictor = task.fit(train_data=train_data, label=label_column, eval_metric='balanced_accuracy', verbosity=4)\n",
    "test_data = task.Dataset(file_path='https://autogluon.s3.amazonaws.com/datasets/diabetes/test.csv')\n",
    "test_data = test_data.head(subsample_size) # subsample data for faster demodisplay(train_data)\n",
    "test_perf = predictor.leaderboard(test_data, silent=True)\n",
    "print(\"Balanced-Accuracy of each model on test data:\")\n",
    "display(test_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see the fine-grained performance of iteratively-trained models during their learning process. Within the call to `fit()`, AutoGluon trained many types of models:\n",
    "\n",
    "- **RandomForestClassifierGini**: Random Forest with [Gini criterion](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) used to select splits in each tree.\n",
    "\n",
    "- **RandomForestClassifierEntr**: Random Forest with [Entropy criterion](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) used to select splits in each tree.\n",
    "\n",
    "- **ExtraTreesClassifierGini**: Extremely-Randomized Trees with [Gini criterion](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier) used to select splits in each tree.\n",
    "\n",
    "- **ExtraTreesClassifierEntr**: Extremely-Randomized Trees with [Entropy criterion](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier) used to select splits in each tree.\n",
    "\n",
    "- **KNeighborsClassifierUnif**: K Nearest Neighbors (KNN) with [uniform weights](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) used to combine labels of nearest-neighbors into prediction.\n",
    "\n",
    "- **KNeighborsClassifierDist**: K Nearest Neighbors with [distance-based weights](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) used to combine labels of nearest-neighbors into prediction.\n",
    "\n",
    "- **LightGBMClassifier**: [LightGBM](https://lightgbm.readthedocs.io/en/latest/) gradient boosted trees with mostly default hyperparameter settings.\n",
    "\n",
    "- **LightGBMClassifierCustom**: [LightGBM](https://lightgbm.readthedocs.io/en/latest/) gradient boosted trees with [custom hyperparameters](https://github.com/awslabs/autogluon/blob/master/autogluon/utils/tabular/ml/models/lgb/hyperparameters/parameters.py) that favor a larger number of larger and more-diverse trees.  \n",
    "\n",
    "- **CatboostClassifier**: [Catboost](https://catboost.ai/) gradient boosted trees.\n",
    "\n",
    "- **NeuralNetClassifier**: Neural network model (implemented in MXNet) whose architecture is printed in the logs above.  \n",
    "\n",
    "Note that traditional ML projects (and many AutoML tools) would treat items like the tree split-criterion or KNN-weighting as decisions/hyperparameters to be optimized. In contrast, AutoGluon simply trains separate models with all options as individual predictors for the weighted ensemble (which increases its diversity). For detailed descriptions of each model type, see: ([Boehmke, 2020: Chapters 8-13](https://bradleyboehmke.github.io/HOML/)).\n",
    "\n",
    "We briefly highlight some properties of certain model types:\n",
    "\n",
    "- LightGBM/CatBoost (and the similar but older XGBoost) are currently the most popular models for tabular data, [almost always used by top Kaggle competitors](https://towardsdatascience.com/four-ways-teams-win-on-kaggle-50e62acb87f4).  [LightGBM](https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree) is one of the fastest gradient-boosting implementations, whereas [CatBoost](https://papers.nips.cc/paper/7898-catboost-unbiased-boosting-with-categorical-features.pdf) offers sophisticated handling of categorical features and is more resilient to overfitting through careful regularization.\n",
    "\n",
    "- Random Forests and Extremely Randomized Trees tend to be robustly performant regardless of their hyperparameter-settings, fast-to-train, but are generally not as accurate as gradient-boosted trees except for smaller datasets. For large datasets, these models can produce undesirably large model files when their maximum tree-depth is uncapped (which usually helps improve their accuracy).\n",
    "\n",
    "- The Neural Network is described in detail in the [**AutoGluon-Tabular** paper](https://arxiv.org/abs/2003.06505), and utilizes [learned embeddings of categorical features](https://arxiv.org/abs/1604.06737), [shortcut-connections](https://arxiv.org/abs/1606.07792), and fully-connected layers with [ReLU activations](https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf), [batch-normalization](https://arxiv.org/abs/1502.03167) and [dropout regularization](http://jmlr.org/papers/v15/srivastava14a.html). These neural networks tend to be among the better models for tabular datasets with many samples, but are often less accurate for datasets with fewer samples or a vast number of features.\n",
    "\n",
    "- K Nearest Neighbors is fast-to-train but slow-for-inference, and typically the least accurate model-type in AutoGluon. Exceptions where KNN outperforms the other model-types include geospatial datasets or simple prediction tasks which can be solved by memorizing the training data.\n",
    "\n",
    "- Even when they are not the most accurate models on a particular dataset, the neural network and KNN model play a key role in AutoGluon. As the rest of the models are all trees, their decision boundaries are sharp and axis-aligned. In constrast, the KNN/neural-network decision boundaries are often far smoother with relatively unconstrained geometry. These models thus provide valuable diversity to the AutoGluon ensemble.\n",
    "\n",
    "<table> \n",
    "    <caption> Decision boundary of fitted models of various types in 3 binary classification tasks: </caption>\n",
    "  <tr>\n",
    "     <td style=\"text-align:center\"> <img src=\"files/images/decisionboundary-input.png\" width=\"100\" height=\"100\"/> </td>\n",
    "    <td style=\"text-align:center\"> <img src=\"files/images/decisionboundary-rf.png\" width=\"100\" height=\"100\"/>  </td>\n",
    "    <td style=\"text-align:center\"> <img src=\"files/images/decisionboundary-nn.png\" width=\"100\" height=\"100\"/> </td>\n",
    "      <td style=\"text-align:center\"> <img src=\"files/images/decisionboundary-knn.png\" width=\"100\" height=\"100\"/>  </td>\n",
    "  </tr>\n",
    "</table>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside the `fit()` call above, AutoGluon performed a random (stratified) split of the data into training and validation sets. A model of each type was trained on the training data (preceded by some model-specific preprocessing applied to a copy of this data passed to the model), and subsequently asked to produce predictions on the validation data. Finally, AutoGluon constructed a weighted ensemble that aggregates the predictions of the individual models in a weighted manner.  Let's take a closer look at this model ensemble:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading: AutogluonModels/ag-20200801_200529/models/weighted_ensemble_k0_l1/model.pkl\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'name': 'weighted_ensemble_k0_l1',\n",
       " 'model_type': 'WeightedEnsembleModel',\n",
       " 'problem_type': 'multiclass',\n",
       " 'eval_metric': 'balanced_accuracy',\n",
       " 'stopping_metric': 'balanced_accuracy',\n",
       " 'fit_time': 0.6886599063873291,\n",
       " 'predict_time': 0.0013511180877685547,\n",
       " 'val_score': 0.5222222222222223,\n",
       " 'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       " 'hyperparameters_fit': {},\n",
       " 'hyperparameters_nondefault': [],\n",
       " 'memory_size': 9055,\n",
       " 'bagged_info': {'child_type': 'GreedyWeightedEnsembleModel',\n",
       "  'num_child_models': 1,\n",
       "  'child_model_names': ['greedy_ensemble'],\n",
       "  '_n_repeats': 1,\n",
       "  '_k_per_n_repeat': [1],\n",
       "  '_random_state': 1,\n",
       "  'low_memory': False,\n",
       "  'bagged_mode': False,\n",
       "  'max_memory_size': 9055,\n",
       "  'min_memory_size': 9055},\n",
       " 'stacker_info': {'num_base_models': 2,\n",
       "  'base_model_names': ['LightGBMClassifier', 'ExtraTreesClassifierEntr'],\n",
       "  'use_orig_features': False},\n",
       " 'children_info': {'greedy_ensemble': {'name': 'greedy_ensemble',\n",
       "   'model_type': 'GreedyWeightedEnsembleModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'balanced_accuracy',\n",
       "   'stopping_metric': 'balanced_accuracy',\n",
       "   'fit_time': 0.6886599063873291,\n",
       "   'predict_time': None,\n",
       "   'val_score': None,\n",
       "   'hyperparameters': {'ensemble_size': 100},\n",
       "   'hyperparameters_fit': {'ensemble_size': 7},\n",
       "   'hyperparameters_nondefault': [],\n",
       "   'memory_size': 6536,\n",
       "   'model_weights': {'LightGBMClassifier': 0.8571428571428571,\n",
       "    'ExtraTreesClassifierEntr': 0.14285714285714285}}}}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ensemble = predictor._trainer.load_model('weighted_ensemble_k0_l1')\n",
    "display(ensemble.get_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `'model_weights'` attribute shows that this ensemble only considers the predictions of some individual models  (all other model-types received weight 0 meaning their predictions are ignored). From `'model_type'`, we see this ensemble is of type `GreedyWeightedEnsembleModel`.\n",
    "\n",
    "For this type of ensemble, the weights are selected via the [Ensemble-Selection algorithm of Caruana et al. (2004)](https://www.cs.cornell.edu/~alexn/papers/shotgun.icml04.revised.rev2.pdf), which is also utilized by some other AutoML tools like [auto-sklearn](https://automl.github.io/auto-sklearn/master/).  The ensemble selection procedure simply uses greedy forward selection to choose the model-weights:\n",
    "\n",
    "0.) Initialize the ensemble so all models start with weight = 0.\n",
    "\n",
    "1.) Consider adding each model to current weighted ensemble (with weight = 1 if this model is not in current ensemble, otherwise increment its weight += 1)\n",
    "\n",
    "2.) Only actually add whichever model contributed the most to ensemble performance on the validation data. The ensemble is evaluated according to the specified evaluation-metric (balanced-accuracy for this particular AutoGluon run), with the current model-weights normalized to sum to 1 when combining model-predictions.\n",
    "\n",
    "3.) Repeatedly iterate (1)-(2), where models may be chosen again even if added in previous rounds.\n",
    "\n",
    "4.) Stop when no more additions can further boost the current ensemble's validation performance.\n",
    "\n",
    "This runs extremely efficiently if we've cached all the predictions of individual models on the validation data.  Because the weights of the ensemble were adaptively selected based on the validation data, be aware that the resulting validation-score of the WeightedEnsemble tends to be more overfit than the individual models' validation-scores.  Nonetheless, this crude procedure for selecting the weights is nice because it:\n",
    "- can be applied with any evaluation-metric (regardless if differentiable/continuous)\n",
    "- produces sparse weights which reduce the ensemble-size\n",
    "- remains more resilient to overfitting than some weighting strategies such as continuous-optimization to find  weights that are globally optimal with respect to validation-score (eg. via regression modeling)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stack Ensembling\n",
    "\n",
    "Stack ensembling is another powerful ensembling technique to additionally grow ensembles and further boost their predictive accuracy ([van Veen et al, 2015](https://mlwave.com/kaggle-ensembling-guide/); [Wolpert, 1992](https://www.sciencedirect.com/science/article/abs/pii/S0893608005800231); [Polley & Van der Laan, 2010](https://biostats.bepress.com/cgi/viewcontent.cgi?article=1269&context=ucbbiostat)). The previously discussed weighted-ensemble combines the predictions of individual models via a simple linear combination, but why should this be the optimal method?\n",
    "\n",
    "In stacking, one instead trains another 'stacker' ML model that takes in the predictions of the individual 'base' models and learns how they should be best combined. This can be achieved simply by training the stacker model on a newly constructed dataset where the features are now all of the base-model predictions (concatenated together, using predicted class-probabilities in classification), and the target-values remain the same as in the original prediction task. The stacker model can rectify shortcomings of the individual base predictions and exploit interactions between them that offer enhanced predictive power ([Van der Laan et al, 2007](https://biostats.bepress.com/cgi/viewcontent.cgi?article=1226&context=ucbbiostat)). But how do we know what type of model the stacker should be?\n",
    "\n",
    "Since we don't know, we can train multiple types of stacker models (just like AutoGluon originally trained multiple types of base models on the original data features). We can once again aggregate these stacker models' predictions via a weighted-ensemble, or we can add another layer of stacker models on top of them. *Multi-layer stacking* feeds the predictions output by the stacker models as inputs to additional higher layer stacker models. Iterating this process in multiple layers has been a winning strategy in prominent prediction competitions ([Koren, 2009](https://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf); [Titericz & Semenov, 2016](https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335)). The particular form of multi-layer stacking used in AutoGluon is shown below, using two stacking layers and $n$ types of base model-types.\n",
    "\n",
    "<img src=\"files/images/multistack.png\" width=\"400\" height=\"400\">\n",
    "\n",
    "By default (when stacking is specified), each stacking layer in AutoGluon simply employs the same model-types (and hyperparameter-values) as the base models trained in the above `fit()`.  AutoGluon stacker models take as input not only the predictions of the models at the previous layer, but also the original data features themselves (inputs to stackers are data features concatenated with lower-layer model predictions). The final stacking layer remains a simple weighted ensemble with weights chosen via Ensemble-Selection as described above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging\n",
    "\n",
    "It is crucial that in `fit()`, stacker models only operate on predictions from base models for datapoints held-out during the base model's training. Otherwise there could be catastrophic overfitting, since the base model could memorize a target-value from the training data and pass this as an input feature to the stacker model.\n",
    "However, using only the validation data for learning the stacker models means these models receive little data, and quickly becomes untenable for stack ensembles with more layers.\n",
    "\n",
    "Instead, AutoGluon can utilize *bagging* (short for \"bootstrap aggregation\") to ensure stacker models are only trained on data held-out from lower layer models, while still being able to learn from the full dataset ([Breiman 1994](https://www.stat.berkeley.edu/~breiman/bagging.pdf); [Breiman 1996](https://statistics.berkeley.edu/sites/default/files/tech-reports/367.pdf)). \n",
    "$k$-fold bagging is a simple ensemble method that reduces variance in the ensemble's resulting predictions. This is achieved by randomly partitioning the data into $k$ disjoint chunks (we stratify based on labels), and subsequently training $k$ different copies of a model with a different data chunk held-out from each copy. AutoGluon bags all models and each model is asked to produce out-of-fold (OOF) predictions on the chunk it did not see during training. As every  datapoint is OOF for one of the bagged model copies, this allows us to obtain OOF predictions from every model for every training example.\n",
    "\n",
    "<img src=\"files/images/bagging.png\" width=\"400\" height=\"400\">\n",
    "\n",
    "In the above data-partioning scheme, we would train one Neural Network for every train/validation split, one Catboost model for every split, etc. (so in total there are $k$ Neural Networks, $k$ Catboost models, etc). We call these $k$ differently-trained copies of the same model-type a *model bag*.\n",
    "\n",
    "At inference-time, new datapoints count as OOF for every model in a $k$-fold bagged ensemble. For a test datapoint, we simply average the predictions of each model in the bag and feed the resulting output into higher-layer stacker models. However there is a subtle train/test mismatch as the input features to stacker models are slightly noisier at training-time than at inference-time. This is because in training, stacker inputs are predictions from a *single* model in the bag, whereas at inference, stacker inputs are averaged predictions across all models in the bag. \n",
    "\n",
    "To mitigate this mismatch, AutoGluon uses *repeated* $k$-fold bagging, in which multiple (randomly chosen) sets of $k$-fold data-partitions and model-bags are maintained. With repeated bagging of multiple $k$-fold bags, each datapoint is remains OOF for 1 model in each $k$-fold bag, meaning we can average the predictions over the multiple models for which this datapoint is OOF even during training. With repeated bagging, we still average predictions over many more models during inference than during training. If we repeat the bagging $n$ times, the stacker input feature variance is $O(n^{-1})$ during training vs. $O((nk)^{-1})$ during inference; without repeated bagging these variances are $O(1)$ vs. $O(k^{-1})$, which is a far greater difference.\n",
    "\n",
    "With multi-layer stacking, AutoGluon's overall training strategy is described in the following algorithm:\n",
    "\n",
    "<img src=\"files/images/trainingalgorithm.png\" width=\"400\" height=\"400\">\n",
    "\n",
    "To produce predictions for obtaining a validation-score, the final set of $\\{ \\hat{Y}_m \\}_{m \\in \\mathcal{M}}$ are combined via a weighted average, with weights chosen via Ensemble-Selection. AutoGluon selects $n$, the number of bagging repeats, based on how much time remains in the user-specified limit after the first round of $k$-fold bagging. If an overall time-limit $T$ is imposed on training, then AutoGluon allows each stacking layer to run for time $T/L$, where $L$ is the number of layers (typically $L=2$ works well in practice). The alloted runtime for training each particular model in a bagging-fold of a stack-layer is divided equally between model-types; model-types that fail to (partially) train within this time are skipped and omitted from the resulting predictor.\n",
    "\n",
    "This design makes the framework highly predictable in its behavior: both the time envelope and failure behavior are well-specified. This approach guarantees that AutoGluon can produce predictions as long as we can train at least one model on one bagging-fold within the allotted time. As we checkpoint intermediate iterations of sequentially-trained models like neural networks and boosted trees, AutoGluon can still produce a model under meager time limits. We additionally anticipate that models may fail while training and just skip to the next model in this event."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In AutoGluon, you can optionally specify the following [arguments to `fit()`](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit):\n",
    "\n",
    "-`num_bagging_folds`: number of bagging folds ($k$ above)\n",
    "\n",
    "-`num_bagging_sets`: number of repeated runs of $k$-fold bagging ($n$ above)\n",
    "\n",
    "-`stack_ensemble_levels`: number of stack layers ($L$ above)\n",
    "\n",
    "However, we recommend users that wish to maximize predictive accuracy via multi-layer stacking instead do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_200607/\n",
      "Beginning AutoGluon training ... Time limit = 120s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_200607/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 1000 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.15s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "Saving AutogluonModels/ag-20200801_200607/learner.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/utils/data/X_train.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/utils/data/y_train.pkl\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l0 ... Training model for up to 59.92s of the 119.85s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/model.pkl\n",
      "\t0.543\t = Validation accuracy score\n",
      "\t7.42s\t = Training runtime\n",
      "\t1.3s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l0 ... Training model for up to 50.68s of the 110.6s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 294 due to low time. Expected time usage reduced from 4.1s -> 4.1s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/model.pkl\n",
      "\t0.533\t = Validation accuracy score\n",
      "\t7.44s\t = Training runtime\n",
      "\t1.27s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l0 ... Training model for up to 41.48s of the 101.4s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 245 due to low time. Expected time usage reduced from 4.0s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 264 due to low time. Expected time usage reduced from 4.1s -> 3.6s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 295 due to low time. Expected time usage reduced from 4.0s -> 4.0s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/model.pkl\n",
      "\t0.509\t = Validation accuracy score\n",
      "\t6.38s\t = Training runtime\n",
      "\t1.25s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l0 ... Training model for up to 32.65s of the 92.57s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 191 due to low time. Expected time usage reduced from 4.1s -> 2.6s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 208 due to low time. Expected time usage reduced from 4.1s -> 2.8s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 234 due to low time. Expected time usage reduced from 4.0s -> 3.1s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 258 due to low time. Expected time usage reduced from 4.0s -> 3.5s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 296 due to low time. Expected time usage reduced from 4.0s -> 3.9s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/model.pkl\n",
      "\t0.515\t = Validation accuracy score\n",
      "\t6.07s\t = Training runtime\n",
      "\t1.27s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierUnif_STACKER_l0 ... Training model for up to 24.31s of the 84.24s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/model.pkl\n",
      "\t0.44\t = Validation accuracy score\n",
      "\t0.12s\t = Training runtime\n",
      "\t1.05s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierDist_STACKER_l0 ... Training model for up to 23.04s of the 82.97s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/model.pkl\n",
      "\t0.467\t = Validation accuracy score\n",
      "\t0.13s\t = Training runtime\n",
      "\t1.06s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: LightGBMClassifier_STACKER_l0 ... Training model for up to 21.78s of the 81.71s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/utils/model_template.pkl\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0.0244444\tvalid_set's multi_error: 0.52\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.52\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.57\n",
      "[50]\ttrain_set's multi_error: 0.0177778\tvalid_set's multi_error: 0.45\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.45\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n",
      "[200]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0.0288889\tvalid_set's multi_error: 0.45\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.43\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.44\n",
      "[50]\ttrain_set's multi_error: 0.0233333\tvalid_set's multi_error: 0.43\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.42\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.4\n",
      "[50]\ttrain_set's multi_error: 0.02\tvalid_set's multi_error: 0.52\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.48\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.48\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0.02\tvalid_set's multi_error: 0.37\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.36\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.4\n",
      "[200]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.38\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0.0155556\tvalid_set's multi_error: 0.52\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.5\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.52\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0.0166667\tvalid_set's multi_error: 0.49\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.48\n",
      "[50]\ttrain_set's multi_error: 0.0188889\tvalid_set's multi_error: 0.46\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.45\n",
      "[50]\ttrain_set's multi_error: 0.02\tvalid_set's multi_error: 0.5\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.49\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.49\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/model.pkl\n",
      "\t0.585\t = Validation accuracy score\n",
      "\t5.59s\t = Training runtime\n",
      "\t0.3s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: CatboostClassifier_STACKER_l0 ... Training model for up to 15.78s of the 75.71s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/utils/model_template.pkl\n",
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.4922222\ttest: 0.5000000\tbest: 0.5000000 (0)\ttotal: 23.1ms\tremaining: 1.13s\n",
      "20:\tlearn: 0.5733333\ttest: 0.6000000\tbest: 0.6200000 (13)\ttotal: 195ms\tremaining: 270ms\n",
      "40:\tlearn: 0.6033333\ttest: 0.5900000\tbest: 0.6200000 (13)\ttotal: 378ms\tremaining: 82.9ms\n",
      "49:\tlearn: 0.6200000\ttest: 0.6200000\tbest: 0.6400000 (46)\ttotal: 463ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.64\n",
      "bestIteration = 46\n",
      "\n",
      "Shrink model to first 47 iterations.\n",
      "0:\tlearn: 0.5977778\ttest: 0.6500000\tbest: 0.6500000 (0)\ttotal: 6.55ms\tremaining: 485ms\n",
      "20:\tlearn: 0.6222222\ttest: 0.6200000\tbest: 0.6500000 (0)\ttotal: 233ms\tremaining: 599ms\n",
      "40:\tlearn: 0.6511111\ttest: 0.6100000\tbest: 0.6500000 (0)\ttotal: 444ms\tremaining: 368ms\n",
      "60:\tlearn: 0.6877778\ttest: 0.6000000\tbest: 0.6500000 (0)\ttotal: 686ms\tremaining: 157ms\n",
      "74:\tlearn: 0.7022222\ttest: 0.6200000\tbest: 0.6500000 (0)\ttotal: 873ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.65\n",
      "bestIteration = 0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5177778\ttest: 0.5600000\tbest: 0.5600000 (0)\ttotal: 10.2ms\tremaining: 501ms\n",
      "20:\tlearn: 0.5733333\ttest: 0.5600000\tbest: 0.5600000 (0)\ttotal: 211ms\tremaining: 292ms\n",
      "40:\tlearn: 0.6188889\ttest: 0.5100000\tbest: 0.5600000 (0)\ttotal: 411ms\tremaining: 90.2ms\n",
      "49:\tlearn: 0.6455556\ttest: 0.5300000\tbest: 0.5600000 (0)\ttotal: 502ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.56\n",
      "bestIteration = 0\n",
      "\n",
      "Shrink model to first 1 iterations.\n",
      "0:\tlearn: 0.5111111\ttest: 0.5600000\tbest: 0.5600000 (0)\ttotal: 6.59ms\tremaining: 442ms\n",
      "20:\tlearn: 0.5977778\ttest: 0.5900000\tbest: 0.5900000 (20)\ttotal: 183ms\tremaining: 410ms\n",
      "40:\tlearn: 0.6333333\ttest: 0.5700000\tbest: 0.5900000 (20)\ttotal: 433ms\tremaining: 285ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60:\tlearn: 0.6700000\ttest: 0.5500000\tbest: 0.5900000 (20)\ttotal: 673ms\tremaining: 77.2ms\n",
      "67:\tlearn: 0.6855556\ttest: 0.5600000\tbest: 0.5900000 (20)\ttotal: 752ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.59\n",
      "bestIteration = 20\n",
      "\n",
      "0:\tlearn: 0.5122222\ttest: 0.5200000\tbest: 0.5200000 (0)\ttotal: 10.2ms\tremaining: 499ms\n",
      "20:\tlearn: 0.5900000\ttest: 0.4900000\tbest: 0.5400000 (3)\ttotal: 239ms\tremaining: 331ms\n",
      "40:\tlearn: 0.6088889\ttest: 0.5200000\tbest: 0.5400000 (3)\ttotal: 480ms\tremaining: 105ms\n",
      "49:\tlearn: 0.6111111\ttest: 0.5100000\tbest: 0.5400000 (3)\ttotal: 610ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.54\n",
      "bestIteration = 3\n",
      "\n",
      "Shrink model to first 4 iterations.\n",
      "0:\tlearn: 0.5311111\ttest: 0.5200000\tbest: 0.5200000 (0)\ttotal: 6.85ms\tremaining: 336ms\n",
      "20:\tlearn: 0.6088889\ttest: 0.5600000\tbest: 0.5600000 (20)\ttotal: 246ms\tremaining: 339ms\n",
      "40:\tlearn: 0.6255556\ttest: 0.5400000\tbest: 0.5600000 (20)\ttotal: 514ms\tremaining: 113ms\n",
      "49:\tlearn: 0.6466667\ttest: 0.5500000\tbest: 0.5600000 (20)\ttotal: 629ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.56\n",
      "bestIteration = 20\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5211111\ttest: 0.5000000\tbest: 0.5000000 (0)\ttotal: 18.7ms\tremaining: 915ms\n",
      "20:\tlearn: 0.5744444\ttest: 0.5200000\tbest: 0.5200000 (2)\ttotal: 245ms\tremaining: 339ms\n",
      "40:\tlearn: 0.6166667\ttest: 0.5200000\tbest: 0.5400000 (22)\ttotal: 481ms\tremaining: 106ms\n",
      "49:\tlearn: 0.6311111\ttest: 0.5200000\tbest: 0.5400000 (22)\ttotal: 588ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.54\n",
      "bestIteration = 22\n",
      "\n",
      "Shrink model to first 23 iterations.\n",
      "0:\tlearn: 0.5744444\ttest: 0.5300000\tbest: 0.5300000 (0)\ttotal: 7.09ms\tremaining: 383ms\n",
      "20:\tlearn: 0.6077778\ttest: 0.5200000\tbest: 0.5300000 (0)\ttotal: 234ms\tremaining: 379ms\n",
      "40:\tlearn: 0.6300000\ttest: 0.5400000\tbest: 0.5600000 (35)\ttotal: 490ms\tremaining: 167ms\n",
      "54:\tlearn: 0.6488889\ttest: 0.5500000\tbest: 0.5600000 (35)\ttotal: 638ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.56\n",
      "bestIteration = 35\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.4977778\ttest: 0.4700000\tbest: 0.4700000 (0)\ttotal: 12ms\tremaining: 588ms\n",
      "20:\tlearn: 0.5888889\ttest: 0.5700000\tbest: 0.6000000 (16)\ttotal: 181ms\tremaining: 250ms\n",
      "40:\tlearn: 0.6255556\ttest: 0.5400000\tbest: 0.6000000 (16)\ttotal: 409ms\tremaining: 89.9ms\n",
      "49:\tlearn: 0.6322222\ttest: 0.5100000\tbest: 0.6000000 (16)\ttotal: 527ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6\n",
      "bestIteration = 16\n",
      "\n",
      "Shrink model to first 17 iterations.\n",
      "0:\tlearn: 0.5688889\ttest: 0.6000000\tbest: 0.6000000 (0)\ttotal: 10.6ms\tremaining: 732ms\n",
      "20:\tlearn: 0.6322222\ttest: 0.5400000\tbest: 0.6000000 (0)\ttotal: 270ms\tremaining: 629ms\n",
      "40:\tlearn: 0.6588889\ttest: 0.5200000\tbest: 0.6000000 (0)\ttotal: 502ms\tremaining: 355ms\n",
      "60:\tlearn: 0.6800000\ttest: 0.5100000\tbest: 0.6000000 (0)\ttotal: 729ms\tremaining: 108ms\n",
      "69:\tlearn: 0.6966667\ttest: 0.5200000\tbest: 0.6000000 (0)\ttotal: 860ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6\n",
      "bestIteration = 0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5055556\ttest: 0.4900000\tbest: 0.4900000 (0)\ttotal: 11.6ms\tremaining: 567ms\n",
      "20:\tlearn: 0.5877778\ttest: 0.5600000\tbest: 0.5600000 (20)\ttotal: 218ms\tremaining: 302ms\n",
      "40:\tlearn: 0.6088889\ttest: 0.5900000\tbest: 0.6200000 (33)\ttotal: 515ms\tremaining: 113ms\n",
      "49:\tlearn: 0.6144444\ttest: 0.5800000\tbest: 0.6200000 (33)\ttotal: 656ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.62\n",
      "bestIteration = 33\n",
      "\n",
      "Shrink model to first 34 iterations.\n",
      "0:\tlearn: 0.6000000\ttest: 0.6400000\tbest: 0.6400000 (0)\ttotal: 8.53ms\tremaining: 418ms\n",
      "20:\tlearn: 0.6177778\ttest: 0.6500000\tbest: 0.6600000 (6)\ttotal: 226ms\tremaining: 312ms\n",
      "40:\tlearn: 0.6377778\ttest: 0.6300000\tbest: 0.6600000 (6)\ttotal: 478ms\tremaining: 105ms\n",
      "49:\tlearn: 0.6588889\ttest: 0.6400000\tbest: 0.6600000 (6)\ttotal: 587ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.66\n",
      "bestIteration = 6\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5244444\ttest: 0.4500000\tbest: 0.4500000 (0)\ttotal: 12.1ms\tremaining: 594ms\n",
      "20:\tlearn: 0.5788889\ttest: 0.5600000\tbest: 0.5700000 (11)\ttotal: 239ms\tremaining: 331ms\n",
      "40:\tlearn: 0.6077778\ttest: 0.5300000\tbest: 0.5700000 (11)\ttotal: 497ms\tremaining: 109ms\n",
      "49:\tlearn: 0.6166667\ttest: 0.5200000\tbest: 0.5700000 (11)\ttotal: 617ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.57\n",
      "bestIteration = 11\n",
      "\n",
      "Shrink model to first 12 iterations.\n",
      "0:\tlearn: 0.5444444\ttest: 0.5700000\tbest: 0.5700000 (0)\ttotal: 7.11ms\tremaining: 405ms\n",
      "20:\tlearn: 0.6011111\ttest: 0.5600000\tbest: 0.5900000 (10)\ttotal: 298ms\tremaining: 524ms\n",
      "40:\tlearn: 0.6422222\ttest: 0.5500000\tbest: 0.5900000 (10)\ttotal: 555ms\tremaining: 230ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "57:\tlearn: 0.6555556\ttest: 0.5700000\tbest: 0.5900000 (10)\ttotal: 778ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.59\n",
      "bestIteration = 10\n",
      "\n",
      "0:\tlearn: 0.4966667\ttest: 0.5100000\tbest: 0.5100000 (0)\ttotal: 11.1ms\tremaining: 542ms\n",
      "20:\tlearn: 0.5966667\ttest: 0.5400000\tbest: 0.5400000 (18)\ttotal: 272ms\tremaining: 376ms\n",
      "40:\tlearn: 0.6322222\ttest: 0.5000000\tbest: 0.5400000 (18)\ttotal: 519ms\tremaining: 114ms\n",
      "49:\tlearn: 0.6344444\ttest: 0.5100000\tbest: 0.5400000 (18)\ttotal: 647ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.54\n",
      "bestIteration = 18\n",
      "\n",
      "Shrink model to first 19 iterations.\n",
      "0:\tlearn: 0.5744444\ttest: 0.5500000\tbest: 0.5500000 (0)\ttotal: 13.1ms\tremaining: 774ms\n",
      "20:\tlearn: 0.6300000\ttest: 0.5100000\tbest: 0.5600000 (6)\ttotal: 243ms\tremaining: 451ms\n",
      "40:\tlearn: 0.6588889\ttest: 0.5300000\tbest: 0.5600000 (6)\ttotal: 507ms\tremaining: 235ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "59:\tlearn: 0.6733333\ttest: 0.5300000\tbest: 0.5600000 (6)\ttotal: 784ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.56\n",
      "bestIteration = 6\n",
      "\n",
      "0:\tlearn: 0.5011111\ttest: 0.4600000\tbest: 0.4600000 (0)\ttotal: 13.9ms\tremaining: 681ms\n",
      "20:\tlearn: 0.5800000\ttest: 0.5000000\tbest: 0.5300000 (13)\ttotal: 237ms\tremaining: 327ms\n",
      "40:\tlearn: 0.6222222\ttest: 0.4900000\tbest: 0.5300000 (13)\ttotal: 569ms\tremaining: 125ms\n",
      "49:\tlearn: 0.6355556\ttest: 0.4800000\tbest: 0.5300000 (13)\ttotal: 736ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.53\n",
      "bestIteration = 13\n",
      "\n",
      "Shrink model to first 14 iterations.\n",
      "0:\tlearn: 0.5722222\ttest: 0.4900000\tbest: 0.4900000 (0)\ttotal: 9.15ms\tremaining: 512ms\n",
      "20:\tlearn: 0.6200000\ttest: 0.4700000\tbest: 0.5100000 (15)\ttotal: 319ms\tremaining: 546ms\n",
      "40:\tlearn: 0.6400000\ttest: 0.4900000\tbest: 0.5100000 (15)\ttotal: 648ms\tremaining: 253ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "56:\tlearn: 0.6577778\ttest: 0.4600000\tbest: 0.5100000 (15)\ttotal: 945ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.51\n",
      "bestIteration = 15\n",
      "\n",
      "0:\tlearn: 0.4888889\ttest: 0.4800000\tbest: 0.4800000 (0)\ttotal: 2.29ms\tremaining: 112ms\n",
      "20:\tlearn: 0.5855556\ttest: 0.5500000\tbest: 0.5700000 (16)\ttotal: 308ms\tremaining: 425ms\n",
      "40:\tlearn: 0.6211111\ttest: 0.5500000\tbest: 0.5700000 (16)\ttotal: 597ms\tremaining: 131ms\n",
      "49:\tlearn: 0.6322222\ttest: 0.5300000\tbest: 0.5700000 (16)\ttotal: 782ms\tremaining: 0us\n",
      "\n",
      "bestTest = 0.57\n",
      "bestIteration = 16\n",
      "\n",
      "Shrink model to first 17 iterations.\n",
      "0:\tlearn: 0.5600000\ttest: 0.5700000\tbest: 0.5700000 (0)\ttotal: 2.56ms\tremaining: 164ms\n",
      "20:\tlearn: 0.6177778\ttest: 0.5200000\tbest: 0.5700000 (0)\ttotal: 432ms\tremaining: 905ms\n",
      "40:\tlearn: 0.6488889\ttest: 0.5400000\tbest: 0.5700000 (0)\ttotal: 752ms\tremaining: 440ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/model.pkl\n",
      "\t0.587\t = Validation accuracy score\n",
      "\t15.25s\t = Training runtime\n",
      "\t0.21s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: NeuralNetClassifier_STACKER_l0 ... Training model for up to 0.28s of the 60.2s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/NeuralNetClassifier_STACKER_l0/utils/model_template.pkl\n",
      "AutoGluon Neural Network infers features are of the following types:\n",
      "{\n",
      "    \"continuous\": [\n",
      "        \"num_lab_procedures\",\n",
      "        \"num_procedures\",\n",
      "        \"num_medications\",\n",
      "        \"number_diagnoses\"\n",
      "    ],\n",
      "    \"skewed\": [\n",
      "        \"time_in_hospital\",\n",
      "        \"number_outpatient\",\n",
      "        \"number_emergency\",\n",
      "        \"number_inpatient\"\n",
      "    ],\n",
      "    \"onehot\": [\n",
      "        \"gender\",\n",
      "        \"repaglinide\",\n",
      "        \"tolbutamide\",\n",
      "        \"pioglitazone\",\n",
      "        \"rosiglitazone\",\n",
      "        \"acarbose\",\n",
      "        \"troglitazone\",\n",
      "        \"tolazamide\",\n",
      "        \"change\",\n",
      "        \"diabetesMed\"\n",
      "    ],\n",
      "    \"embed\": [\n",
      "        \"age\",\n",
      "        \"admission_type_id\",\n",
      "        \"discharge_disposition_id\",\n",
      "        \"admission_source_id\",\n",
      "        \"medical_specialty\",\n",
      "        \"diag_1\",\n",
      "        \"diag_2\",\n",
      "        \"diag_3\",\n",
      "        \"max_glu_serum\",\n",
      "        \"A1Cresult\",\n",
      "        \"metformin\",\n",
      "        \"glimepiride\",\n",
      "        \"glipizide\",\n",
      "        \"glyburide\",\n",
      "        \"insulin\"\n",
      "    ],\n",
      "    \"language\": []\n",
      "}\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60:\tlearn: 0.6755556\ttest: 0.5400000\tbest: 0.5700000 (0)\ttotal: 1.09s\tremaining: 71.7ms\n",
      "64:\tlearn: 0.6800000\ttest: 0.5300000\tbest: 0.5700000 (0)\ttotal: 1.16s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.57\n",
      "bestIteration = 0\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training data for neural network has: 900 examples, 33 features (18 vector, 15 embedding, 0 language)\n",
      "Training neural network for up to 500 epochs...\n",
      "Neural network architecture:\n",
      "EmbedNet(\n",
      "  (numeric_block): NumericBlock(\n",
      "    (body): Dense(None -> 225, Activation(relu))\n",
      "  )\n",
      "  (embed_blocks): HybridSequential(\n",
      "    (0): EmbedBlock(\n",
      "      (body): Embedding(11 -> 6, float32)\n",
      "    )\n",
      "    (1): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (2): EmbedBlock(\n",
      "      (body): Embedding(11 -> 6, float32)\n",
      "    )\n",
      "    (3): EmbedBlock(\n",
      "      (body): Embedding(8 -> 5, float32)\n",
      "    )\n",
      "    (4): EmbedBlock(\n",
      "      (body): Embedding(29 -> 10, float32)\n",
      "    )\n",
      "    (5): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (6): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (7): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (8): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (9): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (10): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (11): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (12): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (13): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (14): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "  )\n",
      "  (output_block): WideAndDeepBlock(\n",
      "    (deep): FeedforwardBlock(\n",
      "      (body): HybridSequential(\n",
      "        (0): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (1): Dropout(p = 0.1, axes=())\n",
      "        (2): Dense(None -> 256, Activation(relu))\n",
      "        (3): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (4): Dropout(p = 0.1, axes=())\n",
      "        (5): Dense(None -> 128, Activation(relu))\n",
      "        (6): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (7): Dropout(p = 0.1, axes=())\n",
      "        (8): Dense(None -> 3, linear)\n",
      "      )\n",
      "    )\n",
      "    (wide): Dense(None -> 3, linear)\n",
      "  )\n",
      ")\n",
      "Epoch 0.  Train loss: 0.6560762, Val accuracy: 0.4\n",
      "\tRan out of time, stopping training early.\n",
      "Best model found in epoch 0. Val accuracy: 0.4\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l0.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Skipping LightGBMClassifierCustom_STACKER_l0 due to lack of time remaining.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Not enough time left to finish repeated k-fold bagging, stopping early ...\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/utils/oof.pkl\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 119.85s of the 59.34s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l1/utils/model_template.pkl\n",
      "Ensemble size: 1\n",
      "Ensemble weights: \n",
      "[1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l1/model.pkl\n",
      "\t0.587\t = Validation accuracy score\n",
      "\t0.55s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/utils/oof.pkl\n",
      "Fitting model: RandomForestClassifierGini_STACKER_l1 ... Training model for up to 58.76s of the 58.74s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l1/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l1/model.pkl\n",
      "\t0.536\t = Validation accuracy score\n",
      "\t9.62s\t = Training runtime\n",
      "\t1.47s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: RandomForestClassifierEntr_STACKER_l1 ... Training model for up to 47.22s of the 47.2s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 275 due to low time. Expected time usage reduced from 4.1s -> 3.8s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/model.pkl\n",
      "\t0.54\t = Validation accuracy score\n",
      "\t9.28s\t = Training runtime\n",
      "\t1.34s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierGini_STACKER_l1 ... Training model for up to 36.19s of the 36.16s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l1/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 211 due to low time. Expected time usage reduced from 4.1s -> 2.9s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 226 due to low time. Expected time usage reduced from 4.1s -> 3.1s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 258 due to low time. Expected time usage reduced from 4.0s -> 3.5s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 292 due to low time. Expected time usage reduced from 4.0s -> 3.9s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l1/model.pkl\n",
      "\t0.531\t = Validation accuracy score\n",
      "\t6.68s\t = Training runtime\n",
      "\t1.32s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: ExtraTreesClassifierEntr_STACKER_l1 ... Training model for up to 27.32s of the 27.3s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l1/utils/model_template.pkl\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 162 due to low time. Expected time usage reduced from 4.0s -> 2.2s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 172 due to low time. Expected time usage reduced from 4.1s -> 2.4s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 193 due to low time. Expected time usage reduced from 4.0s -> 2.6s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 206 due to low time. Expected time usage reduced from 4.2s -> 2.9s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 238 due to low time. Expected time usage reduced from 4.1s -> 3.3s...\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 279 due to low time. Expected time usage reduced from 4.0s -> 3.8s...\n",
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l1/utils/oof.pkl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l1/model.pkl\n",
      "\t0.533\t = Validation accuracy score\n",
      "\t6.62s\t = Training runtime\n",
      "\t1.32s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierUnif_STACKER_l1 ... Training model for up to 18.54s of the 18.52s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l1/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l1/model.pkl\n",
      "\t0.456\t = Validation accuracy score\n",
      "\t0.15s\t = Training runtime\n",
      "\t1.07s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: KNeighborsClassifierDist_STACKER_l1 ... Training model for up to 17.26s of the 17.24s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l1/utils/model_template.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l1/model.pkl\n",
      "\t0.477\t = Validation accuracy score\n",
      "\t0.14s\t = Training runtime\n",
      "\t1.06s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: LightGBMClassifier_STACKER_l1 ... Training model for up to 16.0s of the 15.97s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l1/utils/model_template.pkl\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n",
      "\tRan out of time, early stopping on iteration 50. Best iteration is:\n",
      "\t[18]\ttrain_set's multi_error: 0.0922222\tvalid_set's multi_error: 0.39\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.42\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 31. Best iteration is:\n",
      "\t[2]\ttrain_set's multi_error: 0.342222\tvalid_set's multi_error: 0.46\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n",
      "\tRan out of time, early stopping on iteration 81. Best iteration is:\n",
      "\t[6]\ttrain_set's multi_error: 0.236667\tvalid_set's multi_error: 0.41\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.43\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 147. Best iteration is:\n",
      "\t[26]\ttrain_set's multi_error: 0.0277778\tvalid_set's multi_error: 0.4\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.48\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.49\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 212. Best iteration is:\n",
      "\t[162]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.45\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[200]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.42\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 234. Best iteration is:\n",
      "\t[225]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.39\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[200]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.41\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.48\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.43\n",
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.51\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.44\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.45\n",
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.51\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.53\n",
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.52\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.46\n",
      "[100]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.49\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l1/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l1/model.pkl\n",
      "\t0.583\t = Validation accuracy score\n",
      "\t12.83s\t = Training runtime\n",
      "\t0.39s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: CatboostClassifier_STACKER_l1 ... Training model for up to 2.53s of the 2.51s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l1/utils/model_template.pkl\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\ttrain_set's multi_error: 0\tvalid_set's multi_error: 0.51\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tCatboost model hyperparameters: {'iterations': 10000, 'learning_rate': 0.1, 'random_seed': 0, 'allow_writing_files': False, 'eval_metric': 'Accuracy'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5544444\ttest: 0.6100000\tbest: 0.6100000 (0)\ttotal: 33.1ms\tremaining: 1.62s\n",
      "20:\tlearn: 0.6000000\ttest: 0.5800000\tbest: 0.6100000 (0)\ttotal: 426ms\tremaining: 588ms\n",
      "40:\tlearn: 0.6333333\ttest: 0.6000000\tbest: 0.6100000 (0)\ttotal: 820ms\tremaining: 180ms\n",
      "49:\tlearn: 0.6555556\ttest: 0.6000000\tbest: 0.6100000 (0)\ttotal: 1.02s\tremaining: 0us\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tTime limit exceeded... Skipping CatboostClassifier_STACKER_l1.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: NeuralNetClassifier_STACKER_l1 ... Training model for up to 1.35s of the 1.33s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/NeuralNetClassifier_STACKER_l1/utils/model_template.pkl\n",
      "AutoGluon Neural Network infers features are of the following types:\n",
      "{\n",
      "    \"continuous\": [\n",
      "        \"RandomForestClassifierGini_STACKER_l0_1\",\n",
      "        \"RandomForestClassifierGini_STACKER_l0_2\",\n",
      "        \"RandomForestClassifierEntr_STACKER_l0_1\",\n",
      "        \"RandomForestClassifierEntr_STACKER_l0_2\",\n",
      "        \"ExtraTreesClassifierGini_STACKER_l0_1\",\n",
      "        \"ExtraTreesClassifierGini_STACKER_l0_2\",\n",
      "        \"ExtraTreesClassifierEntr_STACKER_l0_1\",\n",
      "        \"ExtraTreesClassifierEntr_STACKER_l0_2\",\n",
      "        \"KNeighborsClassifierUnif_STACKER_l0_1\",\n",
      "        \"KNeighborsClassifierUnif_STACKER_l0_2\",\n",
      "        \"KNeighborsClassifierDist_STACKER_l0_1\",\n",
      "        \"KNeighborsClassifierDist_STACKER_l0_2\",\n",
      "        \"LightGBMClassifier_STACKER_l0_1\",\n",
      "        \"LightGBMClassifier_STACKER_l0_2\",\n",
      "        \"CatboostClassifier_STACKER_l0_0\",\n",
      "        \"CatboostClassifier_STACKER_l0_1\",\n",
      "        \"CatboostClassifier_STACKER_l0_2\",\n",
      "        \"num_lab_procedures\",\n",
      "        \"num_procedures\",\n",
      "        \"number_diagnoses\"\n",
      "    ],\n",
      "    \"skewed\": [\n",
      "        \"RandomForestClassifierGini_STACKER_l0_0\",\n",
      "        \"RandomForestClassifierEntr_STACKER_l0_0\",\n",
      "        \"ExtraTreesClassifierGini_STACKER_l0_0\",\n",
      "        \"ExtraTreesClassifierEntr_STACKER_l0_0\",\n",
      "        \"KNeighborsClassifierUnif_STACKER_l0_0\",\n",
      "        \"KNeighborsClassifierDist_STACKER_l0_0\",\n",
      "        \"LightGBMClassifier_STACKER_l0_0\",\n",
      "        \"time_in_hospital\",\n",
      "        \"num_medications\",\n",
      "        \"number_outpatient\",\n",
      "        \"number_emergency\",\n",
      "        \"number_inpatient\"\n",
      "    ],\n",
      "    \"onehot\": [\n",
      "        \"gender\",\n",
      "        \"tolbutamide\",\n",
      "        \"pioglitazone\",\n",
      "        \"rosiglitazone\",\n",
      "        \"acarbose\",\n",
      "        \"troglitazone\",\n",
      "        \"tolazamide\",\n",
      "        \"change\",\n",
      "        \"diabetesMed\"\n",
      "    ],\n",
      "    \"embed\": [\n",
      "        \"age\",\n",
      "        \"admission_type_id\",\n",
      "        \"discharge_disposition_id\",\n",
      "        \"admission_source_id\",\n",
      "        \"medical_specialty\",\n",
      "        \"diag_1\",\n",
      "        \"diag_2\",\n",
      "        \"diag_3\",\n",
      "        \"max_glu_serum\",\n",
      "        \"A1Cresult\",\n",
      "        \"metformin\",\n",
      "        \"repaglinide\",\n",
      "        \"glimepiride\",\n",
      "        \"glipizide\",\n",
      "        \"glyburide\",\n",
      "        \"insulin\"\n",
      "    ],\n",
      "    \"language\": []\n",
      "}\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "bestTest = 0.61\n",
      "bestIteration = 0\n",
      "\n",
      "Shrink model to first 1 iterations.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training data for neural network has: 900 examples, 57 features (41 vector, 16 embedding, 0 language)\n",
      "Training neural network for up to 500 epochs...\n",
      "Neural network architecture:\n",
      "EmbedNet(\n",
      "  (numeric_block): NumericBlock(\n",
      "    (body): Dense(None -> 328, Activation(relu))\n",
      "  )\n",
      "  (embed_blocks): HybridSequential(\n",
      "    (0): EmbedBlock(\n",
      "      (body): Embedding(11 -> 6, float32)\n",
      "    )\n",
      "    (1): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (2): EmbedBlock(\n",
      "      (body): Embedding(11 -> 6, float32)\n",
      "    )\n",
      "    (3): EmbedBlock(\n",
      "      (body): Embedding(8 -> 5, float32)\n",
      "    )\n",
      "    (4): EmbedBlock(\n",
      "      (body): Embedding(30 -> 10, float32)\n",
      "    )\n",
      "    (5): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (6): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (7): EmbedBlock(\n",
      "      (body): Embedding(102 -> 21, float32)\n",
      "    )\n",
      "    (8): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (9): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (10): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (11): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (12): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (13): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (14): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "    (15): EmbedBlock(\n",
      "      (body): Embedding(5 -> 3, float32)\n",
      "    )\n",
      "  )\n",
      "  (output_block): WideAndDeepBlock(\n",
      "    (deep): FeedforwardBlock(\n",
      "      (body): HybridSequential(\n",
      "        (0): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (1): Dropout(p = 0.1, axes=())\n",
      "        (2): Dense(None -> 256, Activation(relu))\n",
      "        (3): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (4): Dropout(p = 0.1, axes=())\n",
      "        (5): Dense(None -> 128, Activation(relu))\n",
      "        (6): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
      "        (7): Dropout(p = 0.1, axes=())\n",
      "        (8): Dense(None -> 3, linear)\n",
      "      )\n",
      "    )\n",
      "    (wide): Dense(None -> 3, linear)\n",
      "  )\n",
      ")\n",
      "Epoch 0.  Train loss: 0.61600184, Val accuracy: 0.41\n",
      "\tRan out of time, stopping training early.\n",
      "Best model found in epoch 0. Val accuracy: 0.41\n",
      "\tTime limit exceeded... Skipping NeuralNetClassifier_STACKER_l1.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Fitting model: LightGBMClassifierCustom_STACKER_l1 ... Training model for up to 0.69s of the 0.66s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/LightGBMClassifierCustom_STACKER_l1/utils/model_template.pkl\n",
      "Training Gradient Boosting Model for 10000 rounds...\n",
      "with the following hyperparameter settings:\n",
      "{'num_threads': -1, 'objective': 'multiclass', 'num_classes': 3, 'verbose': -1, 'boosting_type': 'gbdt', 'two_round': True, 'learning_rate': 0.03, 'num_leaves': 128, 'feature_fraction': 0.9, 'min_data_in_leaf': 3, 'seed_value': 0}\n",
      "\tRan out of time, early stopping on iteration 1. Best iteration is:\n",
      "\t[1]\ttrain_set's multi_error: 0.512222\tvalid_set's multi_error: 0.51\n",
      "\tTime limit exceeded... Skipping LightGBMClassifierCustom_STACKER_l1.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Not enough time left to finish repeated k-fold bagging, stopping early ...\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l1/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l1/utils/oof.pkl\n",
      "Fitting model: weighted_ensemble_k0_l2 ... Training model for up to 119.85s of the 0.39s of remaining time.\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l2/utils/model_template.pkl\n",
      "Ensemble size: 1\n",
      "Ensemble weights: \n",
      "[1. 0. 0. 0. 0. 0. 0.]\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l2/utils/oof.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l2/model.pkl\n",
      "\t0.583\t = Validation accuracy score\n",
      "\t0.44s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "Saving AutogluonModels/ag-20200801_200607/models/trainer.pkl\n",
      "AutoGluon training complete, total runtime = 120.12s ...\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/trainer.pkl\n"
     ]
    }
   ],
   "source": [
    "train_data = train_data_full.head(1000)  # need more data to demonstrate automated-stacking\n",
    "predictor_stack = task.fit(train_data=train_data, label=label_column, auto_stack=True, verbosity=3, \n",
    "                           time_limits=120)  # try increasing time_limits to see repeated-bagging in action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `auto_stack` argument tells AutoGluon to automatically select appropriate values of $k, n, L$, which typically are: \n",
    "- $k = 10$ (10-fold bagging)\n",
    "- $n = 20$ (up to 20 bagging repeats, although usually training-time will run out before 20 repeats can be completed)\n",
    "- $L=2$ (two layers of models in stack, followed by a weighted-ensemble to produce final predictions)\n",
    "\n",
    "unless your dataset is small (which is why we used a larger training dataset in the above run).\n",
    "\n",
    "You can see from the above logs that at each stack-level (indicated by suffix **l0** or **l1** at the end of each model's name), AutoGluon is training multiple copies of the models, one for each bagging fold. With the short time-limit imposed above, AutoGluon did not have enough time to train all model-types; to see how it behaves with repeated bagging, try increasing the `time_limits` significantly (recall repeated bagging is only performed to use up any remaining time after one full stack-ensemble with $k$-fold bagging has been trained).\n",
    "\n",
    "Finally, we study the test accuracy of each bagged-model at the various stack-layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading: AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/weighted_ensemble_k0_l2/model.pkl\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.556667</td>\n",
       "      <td>0.515</td>\n",
       "      <td>1.921841</td>\n",
       "      <td>1.269270</td>\n",
       "      <td>6.069274</td>\n",
       "      <td>1.921841</td>\n",
       "      <td>1.269270</td>\n",
       "      <td>6.069274</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CatboostClassifier_STACKER_l0</td>\n",
       "      <td>0.551667</td>\n",
       "      <td>0.587</td>\n",
       "      <td>0.123499</td>\n",
       "      <td>0.214342</td>\n",
       "      <td>15.245624</td>\n",
       "      <td>0.123499</td>\n",
       "      <td>0.214342</td>\n",
       "      <td>15.245624</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>weighted_ensemble_k0_l1</td>\n",
       "      <td>0.551667</td>\n",
       "      <td>0.587</td>\n",
       "      <td>0.136998</td>\n",
       "      <td>0.217539</td>\n",
       "      <td>15.799783</td>\n",
       "      <td>0.013499</td>\n",
       "      <td>0.003197</td>\n",
       "      <td>0.554159</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l0</td>\n",
       "      <td>0.551667</td>\n",
       "      <td>0.533</td>\n",
       "      <td>1.879886</td>\n",
       "      <td>1.266821</td>\n",
       "      <td>7.435487</td>\n",
       "      <td>1.879886</td>\n",
       "      <td>1.266821</td>\n",
       "      <td>7.435487</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l0</td>\n",
       "      <td>0.551667</td>\n",
       "      <td>0.509</td>\n",
       "      <td>1.926960</td>\n",
       "      <td>1.250121</td>\n",
       "      <td>6.377047</td>\n",
       "      <td>1.926960</td>\n",
       "      <td>1.250121</td>\n",
       "      <td>6.377047</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ExtraTreesClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.548333</td>\n",
       "      <td>0.533</td>\n",
       "      <td>12.934189</td>\n",
       "      <td>9.031342</td>\n",
       "      <td>55.014486</td>\n",
       "      <td>2.565373</td>\n",
       "      <td>1.317502</td>\n",
       "      <td>6.615460</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l0</td>\n",
       "      <td>0.543333</td>\n",
       "      <td>0.543</td>\n",
       "      <td>1.883350</td>\n",
       "      <td>1.304344</td>\n",
       "      <td>7.423512</td>\n",
       "      <td>1.883350</td>\n",
       "      <td>1.304344</td>\n",
       "      <td>7.423512</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ExtraTreesClassifierGini_STACKER_l1</td>\n",
       "      <td>0.543333</td>\n",
       "      <td>0.531</td>\n",
       "      <td>12.133149</td>\n",
       "      <td>9.029905</td>\n",
       "      <td>55.079902</td>\n",
       "      <td>1.764333</td>\n",
       "      <td>1.316065</td>\n",
       "      <td>6.680876</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>RandomForestClassifierEntr_STACKER_l1</td>\n",
       "      <td>0.535000</td>\n",
       "      <td>0.540</td>\n",
       "      <td>11.907362</td>\n",
       "      <td>9.057146</td>\n",
       "      <td>57.683584</td>\n",
       "      <td>1.538546</td>\n",
       "      <td>1.343307</td>\n",
       "      <td>9.284559</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>RandomForestClassifierGini_STACKER_l1</td>\n",
       "      <td>0.531667</td>\n",
       "      <td>0.536</td>\n",
       "      <td>11.955314</td>\n",
       "      <td>9.179003</td>\n",
       "      <td>58.015034</td>\n",
       "      <td>1.586498</td>\n",
       "      <td>1.465163</td>\n",
       "      <td>9.616009</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>LightGBMClassifier_STACKER_l0</td>\n",
       "      <td>0.528333</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.412107</td>\n",
       "      <td>0.298448</td>\n",
       "      <td>5.594110</td>\n",
       "      <td>0.412107</td>\n",
       "      <td>0.298448</td>\n",
       "      <td>5.594110</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>LightGBMClassifier_STACKER_l1</td>\n",
       "      <td>0.526667</td>\n",
       "      <td>0.583</td>\n",
       "      <td>10.889278</td>\n",
       "      <td>8.106029</td>\n",
       "      <td>61.232304</td>\n",
       "      <td>0.520462</td>\n",
       "      <td>0.392190</td>\n",
       "      <td>12.833279</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>weighted_ensemble_k0_l2</td>\n",
       "      <td>0.526667</td>\n",
       "      <td>0.583</td>\n",
       "      <td>10.893252</td>\n",
       "      <td>8.108807</td>\n",
       "      <td>61.670950</td>\n",
       "      <td>0.003974</td>\n",
       "      <td>0.002778</td>\n",
       "      <td>0.438646</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l1</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.477</td>\n",
       "      <td>11.474059</td>\n",
       "      <td>8.771866</td>\n",
       "      <td>48.539311</td>\n",
       "      <td>1.105243</td>\n",
       "      <td>1.058026</td>\n",
       "      <td>0.140286</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l0</td>\n",
       "      <td>0.458333</td>\n",
       "      <td>0.440</td>\n",
       "      <td>1.082926</td>\n",
       "      <td>1.050485</td>\n",
       "      <td>0.124062</td>\n",
       "      <td>1.082926</td>\n",
       "      <td>1.050485</td>\n",
       "      <td>0.124062</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>KNeighborsClassifierUnif_STACKER_l1</td>\n",
       "      <td>0.458333</td>\n",
       "      <td>0.456</td>\n",
       "      <td>11.474162</td>\n",
       "      <td>8.781906</td>\n",
       "      <td>48.548296</td>\n",
       "      <td>1.105346</td>\n",
       "      <td>1.068066</td>\n",
       "      <td>0.149271</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>KNeighborsClassifierDist_STACKER_l0</td>\n",
       "      <td>0.451667</td>\n",
       "      <td>0.467</td>\n",
       "      <td>1.138247</td>\n",
       "      <td>1.060008</td>\n",
       "      <td>0.129907</td>\n",
       "      <td>1.138247</td>\n",
       "      <td>1.060008</td>\n",
       "      <td>0.129907</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    model  score_test  score_val  \\\n",
       "0     ExtraTreesClassifierEntr_STACKER_l0    0.556667      0.515   \n",
       "1           CatboostClassifier_STACKER_l0    0.551667      0.587   \n",
       "2                 weighted_ensemble_k0_l1    0.551667      0.587   \n",
       "3   RandomForestClassifierEntr_STACKER_l0    0.551667      0.533   \n",
       "4     ExtraTreesClassifierGini_STACKER_l0    0.551667      0.509   \n",
       "5     ExtraTreesClassifierEntr_STACKER_l1    0.548333      0.533   \n",
       "6   RandomForestClassifierGini_STACKER_l0    0.543333      0.543   \n",
       "7     ExtraTreesClassifierGini_STACKER_l1    0.543333      0.531   \n",
       "8   RandomForestClassifierEntr_STACKER_l1    0.535000      0.540   \n",
       "9   RandomForestClassifierGini_STACKER_l1    0.531667      0.536   \n",
       "10          LightGBMClassifier_STACKER_l0    0.528333      0.585   \n",
       "11          LightGBMClassifier_STACKER_l1    0.526667      0.583   \n",
       "12                weighted_ensemble_k0_l2    0.526667      0.583   \n",
       "13    KNeighborsClassifierDist_STACKER_l1    0.460000      0.477   \n",
       "14    KNeighborsClassifierUnif_STACKER_l0    0.458333      0.440   \n",
       "15    KNeighborsClassifierUnif_STACKER_l1    0.458333      0.456   \n",
       "16    KNeighborsClassifierDist_STACKER_l0    0.451667      0.467   \n",
       "\n",
       "    pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  \\\n",
       "0         1.921841       1.269270   6.069274                 1.921841   \n",
       "1         0.123499       0.214342  15.245624                 0.123499   \n",
       "2         0.136998       0.217539  15.799783                 0.013499   \n",
       "3         1.879886       1.266821   7.435487                 1.879886   \n",
       "4         1.926960       1.250121   6.377047                 1.926960   \n",
       "5        12.934189       9.031342  55.014486                 2.565373   \n",
       "6         1.883350       1.304344   7.423512                 1.883350   \n",
       "7        12.133149       9.029905  55.079902                 1.764333   \n",
       "8        11.907362       9.057146  57.683584                 1.538546   \n",
       "9        11.955314       9.179003  58.015034                 1.586498   \n",
       "10        0.412107       0.298448   5.594110                 0.412107   \n",
       "11       10.889278       8.106029  61.232304                 0.520462   \n",
       "12       10.893252       8.108807  61.670950                 0.003974   \n",
       "13       11.474059       8.771866  48.539311                 1.105243   \n",
       "14        1.082926       1.050485   0.124062                 1.082926   \n",
       "15       11.474162       8.781906  48.548296                 1.105346   \n",
       "16        1.138247       1.060008   0.129907                 1.138247   \n",
       "\n",
       "    pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \n",
       "0                 1.269270           6.069274            0       True  \n",
       "1                 0.214342          15.245624            0       True  \n",
       "2                 0.003197           0.554159            1       True  \n",
       "3                 1.266821           7.435487            0       True  \n",
       "4                 1.250121           6.377047            0       True  \n",
       "5                 1.317502           6.615460            1       True  \n",
       "6                 1.304344           7.423512            0       True  \n",
       "7                 1.316065           6.680876            1       True  \n",
       "8                 1.343307           9.284559            1       True  \n",
       "9                 1.465163           9.616009            1       True  \n",
       "10                0.298448           5.594110            0       True  \n",
       "11                0.392190          12.833279            1       True  \n",
       "12                0.002778           0.438646            2       True  \n",
       "13                1.058026           0.140286            1       True  \n",
       "14                1.050485           0.124062            0       True  \n",
       "15                1.068066           0.149271            1       True  \n",
       "16                1.060008           0.129907            0       True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_perf = predictor_stack.leaderboard(test_data, silent=True)\n",
    "display(test_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that predicting with any of the **l1** models (second layer of the stack ensemble) requires first computing predictions from all **l0** models. Also each \"model\" shown above is actually a bagged ensemble of $k$ differently-trained copies of the same model-type. Let's look closer at one of the **l1** models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_0/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_0/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_1/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_1/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_2/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_2/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_3/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_3/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_4/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_4/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_5/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_5/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_6/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_6/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_7/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_7/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_8/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_8/model.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_9/info.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l1/RandomForestClassifierEntr_fold_9/model.pkl\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'name': 'RandomForestClassifierEntr_STACKER_l1',\n",
       " 'model_type': 'StackerEnsembleModel',\n",
       " 'problem_type': 'multiclass',\n",
       " 'eval_metric': 'accuracy',\n",
       " 'stopping_metric': 'accuracy',\n",
       " 'fit_time': 9.284558773040771,\n",
       " 'predict_time': 1.3433067798614502,\n",
       " 'val_score': 0.54,\n",
       " 'hyperparameters': {'max_models': 25, 'max_models_per_type': 5},\n",
       " 'hyperparameters_fit': {},\n",
       " 'hyperparameters_nondefault': [],\n",
       " 'memory_size': 4924,\n",
       " 'bagged_info': {'child_type': 'RFModel',\n",
       "  'num_child_models': 10,\n",
       "  'child_model_names': ['RandomForestClassifierEntr_fold_0',\n",
       "   'RandomForestClassifierEntr_fold_1',\n",
       "   'RandomForestClassifierEntr_fold_2',\n",
       "   'RandomForestClassifierEntr_fold_3',\n",
       "   'RandomForestClassifierEntr_fold_4',\n",
       "   'RandomForestClassifierEntr_fold_5',\n",
       "   'RandomForestClassifierEntr_fold_6',\n",
       "   'RandomForestClassifierEntr_fold_7',\n",
       "   'RandomForestClassifierEntr_fold_8',\n",
       "   'RandomForestClassifierEntr_fold_9'],\n",
       "  '_n_repeats': 1,\n",
       "  '_k_per_n_repeat': [10],\n",
       "  '_random_state': 1,\n",
       "  'low_memory': True,\n",
       "  'bagged_mode': True,\n",
       "  'max_memory_size': 89003527,\n",
       "  'min_memory_size': 9032422},\n",
       " 'stacker_info': {'num_base_models': 8,\n",
       "  'base_model_names': ['RandomForestClassifierGini_STACKER_l0',\n",
       "   'RandomForestClassifierEntr_STACKER_l0',\n",
       "   'ExtraTreesClassifierGini_STACKER_l0',\n",
       "   'ExtraTreesClassifierEntr_STACKER_l0',\n",
       "   'KNeighborsClassifierUnif_STACKER_l0',\n",
       "   'KNeighborsClassifierDist_STACKER_l0',\n",
       "   'LightGBMClassifier_STACKER_l0',\n",
       "   'CatboostClassifier_STACKER_l0'],\n",
       "  'use_orig_features': True},\n",
       " 'children_info': {'RandomForestClassifierEntr_fold_0': {'name': 'RandomForestClassifierEntr_fold_0',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.9823169708251953,\n",
       "   'predict_time': 0.13515281677246094,\n",
       "   'val_score': 0.54,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 275},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8268944},\n",
       "  'RandomForestClassifierEntr_fold_1': {'name': 'RandomForestClassifierEntr_fold_1',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.9494080543518066,\n",
       "   'predict_time': 0.12700986862182617,\n",
       "   'val_score': 0.55,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8999329},\n",
       "  'RandomForestClassifierEntr_fold_2': {'name': 'RandomForestClassifierEntr_fold_2',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.9623019695281982,\n",
       "   'predict_time': 0.16666007041931152,\n",
       "   'val_score': 0.55,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 9027498},\n",
       "  'RandomForestClassifierEntr_fold_3': {'name': 'RandomForestClassifierEntr_fold_3',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.9953598976135254,\n",
       "   'predict_time': 0.1283869743347168,\n",
       "   'val_score': 0.55,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8955329},\n",
       "  'RandomForestClassifierEntr_fold_4': {'name': 'RandomForestClassifierEntr_fold_4',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.8493170738220215,\n",
       "   'predict_time': 0.13339686393737793,\n",
       "   'val_score': 0.5,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8922369},\n",
       "  'RandomForestClassifierEntr_fold_5': {'name': 'RandomForestClassifierEntr_fold_5',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.8676481246948242,\n",
       "   'predict_time': 0.12713193893432617,\n",
       "   'val_score': 0.61,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 9015498},\n",
       "  'RandomForestClassifierEntr_fold_6': {'name': 'RandomForestClassifierEntr_fold_6',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.8562788963317871,\n",
       "   'predict_time': 0.127593994140625,\n",
       "   'val_score': 0.59,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8954529},\n",
       "  'RandomForestClassifierEntr_fold_7': {'name': 'RandomForestClassifierEntr_fold_7',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.998931884765625,\n",
       "   'predict_time': 0.13928914070129395,\n",
       "   'val_score': 0.5,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8927329},\n",
       "  'RandomForestClassifierEntr_fold_8': {'name': 'RandomForestClassifierEntr_fold_8',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.9530348777770996,\n",
       "   'predict_time': 0.12943601608276367,\n",
       "   'val_score': 0.52,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8992449},\n",
       "  'RandomForestClassifierEntr_fold_9': {'name': 'RandomForestClassifierEntr_fold_9',\n",
       "   'model_type': 'RFModel',\n",
       "   'problem_type': 'multiclass',\n",
       "   'eval_metric': 'accuracy',\n",
       "   'stopping_metric': 'accuracy',\n",
       "   'fit_time': 0.8699610233306885,\n",
       "   'predict_time': 0.12924909591674805,\n",
       "   'val_score': 0.49,\n",
       "   'hyperparameters': {'n_estimators': 300,\n",
       "    'n_jobs': -1,\n",
       "    'criterion': 'entropy'},\n",
       "   'hyperparameters_fit': {'n_estimators': 300},\n",
       "   'hyperparameters_nondefault': ['criterion'],\n",
       "   'memory_size': 8935329}}}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rf_stacker = predictor_stack._trainer.load_model('RandomForestClassifierEntr_STACKER_l1')\n",
    "display(rf_stacker.get_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see this `StackerEnsembleModel` object actually contains 10 different **RandomForestClassifierEntr** models in its bag (listed under attribute `'child_model_names'`). Attribute `'base_model_names'` shows this stacker model takes as input the predictions of all **l0** base models that managed to train within the alloted time-limits. Here's what the processed input features for this particular model look like: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading: AutogluonModels/ag-20200801_200607/utils/data/X_train.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/RandomForestClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierGini_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/ExtraTreesClassifierEntr_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierUnif_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/KNeighborsClassifierDist_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/LightGBMClassifier_STACKER_l0/utils/oof.pkl\n",
      "Loading: AutogluonModels/ag-20200801_200607/models/CatboostClassifier_STACKER_l0/utils/oof.pkl\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RandomForestClassifierGini_STACKER_l0_0</th>\n",
       "      <th>RandomForestClassifierGini_STACKER_l0_1</th>\n",
       "      <th>RandomForestClassifierGini_STACKER_l0_2</th>\n",
       "      <th>RandomForestClassifierEntr_STACKER_l0_0</th>\n",
       "      <th>RandomForestClassifierEntr_STACKER_l0_1</th>\n",
       "      <th>RandomForestClassifierEntr_STACKER_l0_2</th>\n",
       "      <th>ExtraTreesClassifierGini_STACKER_l0_0</th>\n",
       "      <th>ExtraTreesClassifierGini_STACKER_l0_1</th>\n",
       "      <th>ExtraTreesClassifierGini_STACKER_l0_2</th>\n",
       "      <th>ExtraTreesClassifierEntr_STACKER_l0_0</th>\n",
       "      <th>...</th>\n",
       "      <th>glyburide</th>\n",
       "      <th>tolbutamide</th>\n",
       "      <th>pioglitazone</th>\n",
       "      <th>rosiglitazone</th>\n",
       "      <th>acarbose</th>\n",
       "      <th>troglitazone</th>\n",
       "      <th>tolazamide</th>\n",
       "      <th>insulin</th>\n",
       "      <th>change</th>\n",
       "      <th>diabetesMed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.210000</td>\n",
       "      <td>0.780000</td>\n",
       "      <td>0.006803</td>\n",
       "      <td>0.261905</td>\n",
       "      <td>0.731293</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.191837</td>\n",
       "      <td>0.808163</td>\n",
       "      <td>0.015707</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.506667</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.383333</td>\n",
       "      <td>0.416667</td>\n",
       "      <td>0.126667</td>\n",
       "      <td>0.273333</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.131783</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.206667</td>\n",
       "      <td>0.246667</td>\n",
       "      <td>0.546667</td>\n",
       "      <td>0.236667</td>\n",
       "      <td>0.210000</td>\n",
       "      <td>0.553333</td>\n",
       "      <td>0.263333</td>\n",
       "      <td>0.276667</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.220000</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.433333</td>\n",
       "      <td>0.406667</td>\n",
       "      <td>0.136667</td>\n",
       "      <td>0.383333</td>\n",
       "      <td>0.480000</td>\n",
       "      <td>0.135593</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.464407</td>\n",
       "      <td>0.158120</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.380000</td>\n",
       "      <td>0.520000</td>\n",
       "      <td>0.076667</td>\n",
       "      <td>0.390000</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.063333</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>0.353333</td>\n",
       "      <td>0.093333</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>0.243333</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.286667</td>\n",
       "      <td>0.273333</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.256667</td>\n",
       "      <td>0.256667</td>\n",
       "      <td>0.396667</td>\n",
       "      <td>0.346667</td>\n",
       "      <td>0.302326</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>0.043333</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.706667</td>\n",
       "      <td>0.003401</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.710884</td>\n",
       "      <td>0.016327</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>0.697959</td>\n",
       "      <td>0.005236</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.386667</td>\n",
       "      <td>0.483333</td>\n",
       "      <td>0.153333</td>\n",
       "      <td>0.373333</td>\n",
       "      <td>0.473333</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.320000</td>\n",
       "      <td>0.480000</td>\n",
       "      <td>0.205426</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>0.116667</td>\n",
       "      <td>0.370000</td>\n",
       "      <td>0.513333</td>\n",
       "      <td>0.116667</td>\n",
       "      <td>0.366667</td>\n",
       "      <td>0.516667</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.373333</td>\n",
       "      <td>0.476667</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.080000</td>\n",
       "      <td>0.196667</td>\n",
       "      <td>0.723333</td>\n",
       "      <td>0.070000</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.065891</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     RandomForestClassifierGini_STACKER_l0_0  \\\n",
       "0                                   0.010000   \n",
       "1                                   0.160000   \n",
       "2                                   0.206667   \n",
       "3                                   0.160000   \n",
       "4                                   0.100000   \n",
       "..                                       ...   \n",
       "995                                 0.243333   \n",
       "996                                 0.043333   \n",
       "997                                 0.130000   \n",
       "998                                 0.116667   \n",
       "999                                 0.133333   \n",
       "\n",
       "     RandomForestClassifierGini_STACKER_l0_1  \\\n",
       "0                                   0.210000   \n",
       "1                                   0.333333   \n",
       "2                                   0.246667   \n",
       "3                                   0.433333   \n",
       "4                                   0.380000   \n",
       "..                                       ...   \n",
       "995                                 0.470000   \n",
       "996                                 0.250000   \n",
       "997                                 0.386667   \n",
       "998                                 0.370000   \n",
       "999                                 0.166667   \n",
       "\n",
       "     RandomForestClassifierGini_STACKER_l0_2  \\\n",
       "0                                   0.780000   \n",
       "1                                   0.506667   \n",
       "2                                   0.546667   \n",
       "3                                   0.406667   \n",
       "4                                   0.520000   \n",
       "..                                       ...   \n",
       "995                                 0.286667   \n",
       "996                                 0.706667   \n",
       "997                                 0.483333   \n",
       "998                                 0.513333   \n",
       "999                                 0.700000   \n",
       "\n",
       "     RandomForestClassifierEntr_STACKER_l0_0  \\\n",
       "0                                   0.006803   \n",
       "1                                   0.200000   \n",
       "2                                   0.236667   \n",
       "3                                   0.136667   \n",
       "4                                   0.076667   \n",
       "..                                       ...   \n",
       "995                                 0.273333   \n",
       "996                                 0.003401   \n",
       "997                                 0.153333   \n",
       "998                                 0.116667   \n",
       "999                                 0.080000   \n",
       "\n",
       "     RandomForestClassifierEntr_STACKER_l0_1  \\\n",
       "0                                   0.261905   \n",
       "1                                   0.383333   \n",
       "2                                   0.210000   \n",
       "3                                   0.383333   \n",
       "4                                   0.390000   \n",
       "..                                       ...   \n",
       "995                                 0.470000   \n",
       "996                                 0.285714   \n",
       "997                                 0.373333   \n",
       "998                                 0.366667   \n",
       "999                                 0.196667   \n",
       "\n",
       "     RandomForestClassifierEntr_STACKER_l0_2  \\\n",
       "0                                   0.731293   \n",
       "1                                   0.416667   \n",
       "2                                   0.553333   \n",
       "3                                   0.480000   \n",
       "4                                   0.533333   \n",
       "..                                       ...   \n",
       "995                                 0.256667   \n",
       "996                                 0.710884   \n",
       "997                                 0.473333   \n",
       "998                                 0.516667   \n",
       "999                                 0.723333   \n",
       "\n",
       "     ExtraTreesClassifierGini_STACKER_l0_0  \\\n",
       "0                                 0.000000   \n",
       "1                                 0.126667   \n",
       "2                                 0.263333   \n",
       "3                                 0.135593   \n",
       "4                                 0.063333   \n",
       "..                                     ...   \n",
       "995                               0.256667   \n",
       "996                               0.016327   \n",
       "997                               0.200000   \n",
       "998                               0.150000   \n",
       "999                               0.070000   \n",
       "\n",
       "     ExtraTreesClassifierGini_STACKER_l0_1  \\\n",
       "0                                 0.191837   \n",
       "1                                 0.273333   \n",
       "2                                 0.276667   \n",
       "3                                 0.400000   \n",
       "4                                 0.583333   \n",
       "..                                     ...   \n",
       "995                               0.396667   \n",
       "996                               0.285714   \n",
       "997                               0.320000   \n",
       "998                               0.373333   \n",
       "999                               0.130000   \n",
       "\n",
       "     ExtraTreesClassifierGini_STACKER_l0_2  \\\n",
       "0                                 0.808163   \n",
       "1                                 0.600000   \n",
       "2                                 0.460000   \n",
       "3                                 0.464407   \n",
       "4                                 0.353333   \n",
       "..                                     ...   \n",
       "995                               0.346667   \n",
       "996                               0.697959   \n",
       "997                               0.480000   \n",
       "998                               0.476667   \n",
       "999                               0.800000   \n",
       "\n",
       "     ExtraTreesClassifierEntr_STACKER_l0_0  ...  glyburide  tolbutamide  \\\n",
       "0                                 0.015707  ...          1            0   \n",
       "1                                 0.131783  ...          1            0   \n",
       "2                                 0.220000  ...          1            0   \n",
       "3                                 0.158120  ...          1            0   \n",
       "4                                 0.093333  ...          1            0   \n",
       "..                                     ...  ...        ...          ...   \n",
       "995                               0.302326  ...          1            0   \n",
       "996                               0.005236  ...          1            0   \n",
       "997                               0.205426  ...          2            0   \n",
       "998                               0.160000  ...          1            0   \n",
       "999                               0.065891  ...          1            0   \n",
       "\n",
       "     pioglitazone  rosiglitazone  acarbose  troglitazone  tolazamide  insulin  \\\n",
       "0               0              0         0             0           0        1   \n",
       "1               0              0         0             0           0        3   \n",
       "2               0              0         0             0           0        1   \n",
       "3               0              0         0             0           0        3   \n",
       "4               0              0         0             0           0        2   \n",
       "..            ...            ...       ...           ...         ...      ...   \n",
       "995             0              0         0             0           0        2   \n",
       "996             0              0         0             0           0        2   \n",
       "997             0              0         0             0           0        2   \n",
       "998             0              0         0             0           0        0   \n",
       "999             0              0         0             0           0        1   \n",
       "\n",
       "     change  diabetesMed  \n",
       "0         1            0  \n",
       "1         0            1  \n",
       "2         1            1  \n",
       "3         0            1  \n",
       "4         0            1  \n",
       "..      ...          ...  \n",
       "995       1            1  \n",
       "996       1            1  \n",
       "997       0            1  \n",
       "998       0            1  \n",
       "999       0            1  \n",
       "\n",
       "[1000 rows x 57 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor_stack.transform_features(model='RandomForestClassifierEntr_STACKER_l1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall these input features are the concatenation of the original data features (after model-agnostic preprocessing) and the predicted class-probabilities output by each trained **l0** base model.  Since our prediction task here is 3-class classification, there are 3 class-probabilities from each base model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## What about hyperparameter tuning? \n",
    "\n",
    "It is well known among competitive data scientists that creating model ensembles via stacking/bagging tends to outperform other meta-approaches to boost models' accuracy such as *hyperparameter optimization* (HPO). This is particularly true under training-time constraints, since HPO often expends significant compute evaluating poor hyperparameter-configurations no reasonable data scientist would consider. When increasing the alloted time-budget, one would expect an AutoML system to continuously improve. However, this is not always the case with HPO methods, which can begin to heavily overfit their validation data if their search is allowed to try too many hyperparameter-configurations. Given additional runtime, AutoGluon's approach\n",
    "(for tabular data with `auto_stack` specified) is to instead train more model-types with more repeated bagging models to keep growing its ensemble. This more reliably improves predictive performance with additional runtime and resists  overfitting much better than extensive HPO [(Erickson et al, 2020)](https://arxiv.org/abs/2003.06505). \n",
    "\n",
    "That said, the accuracy of AutoGluon-Tabular can often be further improved through hyperparameter-tuning, which you can easily utilize via the [`hyperparameter_tune` argument of `fit()`](https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.TabularPrediction.fit) (both with or without stack-ensembling). Below we show an example only training and tuning only LightGBM models, with some user-specified [hyperparameter](https://lightgbm.readthedocs.io/en/latest/Parameters.html) search-spaces for demonstration purposes (you don't need to provide these for HPO, as AutoGluon has default hyperparameter search-spaces it will consider for every model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: `hyperparameter_tune=True` is currently experimental and may cause the process to hang. Setting `auto_stack=True` instead is recommended to achieve maximum quality models.\n",
      "No output_directory specified. Models will be saved in: AutogluonModels/ag-20200801_200828/\n",
      "Beginning AutoGluon training ... Time limit = 100s\n",
      "AutoGluon will save models to AutogluonModels/ag-20200801_200828/\n",
      "AutoGluon Version:  0.0.13b20200731\n",
      "Train Data Rows:    600\n",
      "Train Data Columns: 47\n",
      "Preprocessing data ...\n",
      "Here are the 3 unique label values in your data:  ['NO', '>30', '<30']\n",
      "AutoGluon infers your prediction problem is: multiclass  (because dtype of label-column == object).\n",
      "If this is wrong, please specify `problem_type` argument in fit() instead (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "\n",
      "Train Data Class Count: 3\n",
      "Feature Generator processed 600 data points with 33 features\n",
      "Original Features (raw dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat64 features: 1\n",
      "\tint64 features: 7\n",
      "Original Features (inferred dtypes):\n",
      "\tobject features: 25\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "Generated Features (special dtypes):\n",
      "Processed Features (raw dtypes):\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "Processed Features:\n",
      "\tfloat features: 1\n",
      "\tint features: 7\n",
      "\tcategory features: 25\n",
      "\tData preprocessing and feature engineering runtime = 0.15s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: accuracy\n",
      "To change this, specify the eval_metric argument of fit()\n",
      "AutoGluon will early stop models using evaluation metric: accuracy\n",
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n",
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 1000\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "599819b6316a48e1b06057d6d94e71dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Time out (secs) is 90.0\n",
      "\tRan out of time, early stopping on iteration 221. Best iteration is:\n",
      "\t[147]\ttrain_set's multi_logloss: 0.269456\ttrain_set's multi_error: 0.00833333\tvalid_set's multi_logloss: 0.878179\tvalid_set's multi_error: 0.358333\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t0.6417\t = Validation accuracy score\n",
      "\t4.27s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6333\t = Validation accuracy score\n",
      "\t1.69s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t1.16s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "\t0.5667\t = Validation accuracy score\n",
      "\t1.46s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t1.99s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.6333\t = Validation accuracy score\n",
      "\t0.95s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.625\t = Validation accuracy score\n",
      "\t2.33s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t1.85s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6333\t = Validation accuracy score\n",
      "\t2.07s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t3.27s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.6417\t = Validation accuracy score\n",
      "\t2.77s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t2.07s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5833\t = Validation accuracy score\n",
      "\t0.99s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t1.39s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t2.49s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5833\t = Validation accuracy score\n",
      "\t3.45s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t1.62s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t2.91s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6\t = Validation accuracy score\n",
      "\t1.88s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t3.06s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6333\t = Validation accuracy score\n",
      "\t2.44s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t2.05s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t1.45s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "\t0.6167\t = Validation accuracy score\n",
      "\t3.01s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6083\t = Validation accuracy score\n",
      "\t2.33s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.575\t = Validation accuracy score\n",
      "\t1.93s\t = Training runtime\n",
      "\t0.07s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t2.5s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6417\t = Validation accuracy score\n",
      "\t5.29s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.625\t = Validation accuracy score\n",
      "\t3.98s\t = Training runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "\t0.5917\t = Validation accuracy score\n",
      "\t1.32s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.6417\t = Validation accuracy score\n",
      "\t3.14s\t = Training runtime\n",
      "\t0.07s\t = Validation runtime\n",
      "\t0.6417\t = Validation accuracy score\n",
      "\t5.23s\t = Training runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "\t0.6167\t = Validation accuracy score\n",
      "\t2.45s\t = Training runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "\t0.5833\t = Validation accuracy score\n",
      "\t1.21s\t = Training runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 99.85s of the 6.3s of remaining time.\n",
      "\t0.6667\t = Validation accuracy score\n",
      "\t0.23s\t = Training runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 94.25s ...\n"
     ]
    }
   ],
   "source": [
    "import autogluon as ag\n",
    "train_data = train_data_full.head(600)  # subsample for faster demo\n",
    "\n",
    "gbm_options = { # specifies non-default hyperparameter values/search-spaces for LightGBM models (unspecified hyperparameters may still be tuned within default search-space)\n",
    "    'num_leaves': ag.space.Int(lower=26, upper=66, default=36), # number of leaves in trees (integer hyperparameter)\n",
    "    'learning_rate': ag.space.Real(1e-2, 0.5, default=0.05, log=True), # learning rate used in training (real-valued hyperparameter searched on log-scale)\n",
    "    'boosting_type': ag.space.Categorical('gbdt', 'goss') # hyperparameter of discrete choices (first element is the default-value that gets tried first).\n",
    "}\n",
    "\n",
    "predictor_hpo = task.fit(train_data=train_data, label=label_column, hyperparameter_tune=True,\n",
    "                         hyperparameters={'GBM':gbm_options}, time_limits=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we examine the performance of the models trained under each hyperparameter-configuration considered in the search. Some of AutoGluon's internally-tracked performance-numbers may be displayed as negative, as these are error-rates whose sign was flipped since AutoGluon internally assumes higher-values are better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Summary of fit() ***\n",
      "Estimated performance of each model:\n",
      "                          model  score_val  pred_time_val   fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer\n",
      "0       weighted_ensemble_k0_l1   0.666667       0.197409  12.870424                0.001155           0.233449            1       True\n",
      "1    LightGBMClassifier/trial_0   0.641667       0.047526   4.267272                0.047526           4.267272            0       True\n",
      "2   LightGBMClassifier/trial_18   0.641667       0.049674   2.772787                0.049674           2.772787            0       True\n",
      "3   LightGBMClassifier/trial_33   0.641667       0.051146   5.293578                0.051146           5.293578            0       True\n",
      "4    LightGBMClassifier/trial_6   0.641667       0.067601   3.142998                0.067601           3.142998            0       True\n",
      "5    LightGBMClassifier/trial_7   0.641667       0.081126   5.226705                0.081126           5.226705            0       True\n",
      "6    LightGBMClassifier/trial_1   0.633333       0.043306   1.685692                0.043306           1.685692            0       True\n",
      "7   LightGBMClassifier/trial_13   0.633333       0.045684   0.947038                0.045684           0.947038            0       True\n",
      "8   LightGBMClassifier/trial_16   0.633333       0.046900   2.067910                0.046900           2.067910            0       True\n",
      "9   LightGBMClassifier/trial_27   0.633333       0.051131   2.442108                0.051131           2.442108            0       True\n",
      "10  LightGBMClassifier/trial_14   0.625000       0.055795   2.325474                0.055795           2.325474            0       True\n",
      "11   LightGBMClassifier/trial_4   0.625000       0.058177   3.984898                0.058177           3.984898            0       True\n",
      "12   LightGBMClassifier/trial_8   0.616667       0.048761   2.454168                0.048761           2.454168            0       True\n",
      "13   LightGBMClassifier/trial_3   0.616667       0.051869   3.006539                0.051869           3.006539            0       True\n",
      "14  LightGBMClassifier/trial_10   0.608333       0.040342   1.157142                0.040342           1.157142            0       True\n",
      "15  LightGBMClassifier/trial_29   0.608333       0.041040   1.454215                0.041040           1.454215            0       True\n",
      "16  LightGBMClassifier/trial_30   0.608333       0.047175   2.330844                0.047175           2.330844            0       True\n",
      "17  LightGBMClassifier/trial_21   0.608333       0.052969   2.490205                0.052969           2.490205            0       True\n",
      "18  LightGBMClassifier/trial_25   0.600000       0.042695   1.877959                0.042695           1.877959            0       True\n",
      "19  LightGBMClassifier/trial_19   0.600000       0.047043   2.066163                0.047043           2.066163            0       True\n",
      "20  LightGBMClassifier/trial_20   0.600000       0.047096   1.386516                0.047096           1.386516            0       True\n",
      "21  LightGBMClassifier/trial_15   0.600000       0.048483   1.851806                0.048483           1.851806            0       True\n",
      "22  LightGBMClassifier/trial_12   0.600000       0.061794   1.990369                0.061794           1.990369            0       True\n",
      "23   LightGBMClassifier/trial_5   0.591667       0.047047   1.320977                0.047047           1.320977            0       True\n",
      "24  LightGBMClassifier/trial_28   0.591667       0.048045   2.053825                0.048045           2.053825            0       True\n",
      "25  LightGBMClassifier/trial_23   0.591667       0.049108   1.623514                0.049108           1.623514            0       True\n",
      "26  LightGBMClassifier/trial_32   0.591667       0.049713   2.504859                0.049713           2.504859            0       True\n",
      "27  LightGBMClassifier/trial_26   0.591667       0.050431   3.064881                0.050431           3.064881            0       True\n",
      "28  LightGBMClassifier/trial_24   0.591667       0.051741   2.911462                0.051741           2.911462            0       True\n",
      "29  LightGBMClassifier/trial_17   0.591667       0.055113   3.269295                0.055113           3.269295            0       True\n",
      "30   LightGBMClassifier/trial_9   0.583333       0.043964   1.214359                0.043964           1.214359            0       True\n",
      "31   LightGBMClassifier/trial_2   0.583333       0.057978   0.987094                0.057978           0.987094            0       True\n",
      "32  LightGBMClassifier/trial_22   0.583333       0.061381   3.451736                0.061381           3.451736            0       True\n",
      "33  LightGBMClassifier/trial_31   0.575000       0.065793   1.928291                0.065793           1.928291            0       True\n",
      "34  LightGBMClassifier/trial_11   0.566667       0.050352   1.455196                0.050352           1.455196            0       True\n",
      "Number of models trained: 35\n",
      "Types of models trained:\n",
      "{'LGBModel', 'WeightedEnsembleModel'}\n",
      "Bagging used: False \n",
      "Stack-ensembling used: False \n",
      "Hyperparameter-tuning used: True \n",
      "User-specified hyperparameters:\n",
      "{'default': {'GBM': [{'num_leaves': Int: lower=26, upper=66, 'learning_rate': Real: lower=0.01, upper=0.5, 'boosting_type': Categorical['gbdt', 'goss']}]}}\n",
      "Plot summary of models saved to file: AutogluonModels/ag-20200801_200828/SummaryOfModels.html\n",
      "Plot summary of models saved to file: AutogluonModels/ag-20200801_200828/LightGBMClassifier_HPOmodelsummary.html\n",
      "Plot summary of models saved to file: LightGBMClassifier_HPOmodelsummary.html\n",
      "Plot of HPO performance saved to file: AutogluonModels/ag-20200801_200828/LightGBMClassifier_HPOperformanceVStrials.png\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAltklEQVR4nO3deZwcdZ3/8dc7dwggQWIIR8hPLlE2HGZFWHBRgscKxhNwPcIqIrK66y64oHjguah4oK5HRCVeCCKXAq5JkMMLCAhJEEOAheUIAeIi0A3TmZnP74/6Dnbanpme7qrpme738/GYx3RVfavq01Uz/env91tVX0UEZmZmzZjQ7gDMzGz8chIxM7OmOYmYmVnTnETMzKxpTiJmZtY0JxEzM2uak4i1RNIVkhY3WPZuSQub2Me7JG2Q9ISkZ448yob3c4iktQ2WPVTSfUXFMhokXSXpuIK2PTedr4lperakayQ9Lulzkj4g6ewi9p03SedI+kSDZZv6Gx/PnETGgXp/mJKOlfSrmjJPpn/cDekPf8uq5UdIul5SSdJGST+QtNMQ+zxd0veHiy0iXhERS5t9b8ORNBn4PPDSiNgyIjbmsM26/+gRcW1E7Nnq9tM+6n7wSDpG0nXpPDyUXp8oSVXrVdJ5fFzSjZL+vmr9YyWFpC/UbHdRmn9O1bwp6TyuS/u7W9K3Jc3L4z0OJSL+N52vvjTreOARYOuIOCkiPhURhSQwG11OIp3lyIjYEtgfWAB8EEDS64EfAl8EtgOeB/QAv5I0s5kdKTMafz+zgWnArSNdcRRjbIikk4CzgM8C25O9txOAvwOmVBX9TDqPWwNfAy4c+Eaf3AkcJWlS1bzFwO01u7wAeBXwj8AzgH2AG4HD8npPI7AL8Ido8e7msXZOzUmkI0XE/cAVwN7pG+7ngE9ExA8j4smIeBA4DngC+Lfa9SW9HPgAcHT6RnxLmn+VpE9K+jVQBp5d3SQiaVdJV6aaziOptrNNvRglvUDSSkmPpZrT5+uU2QMYaF56VNKVaf5Bkm6Q9Of0+6Cqdf4qxkaPW20TlaT9Jf0+1Qh+LOm82tqFpJNSjWK9pH9K844H3gT8Rzp+P5X0DOBjwIkRcUFEPB6Z30fEmyKipzae9IH7Q2BbsoQz4EFgNfCytL9tgYOAS6viWggcDiyKiBsiojci/hwR/xUR36rz3oc8d5JOkXR/OhZrJR2W5tc9j5LmpZrRpFQ7Wlx1PBbW1nQlvVDSbyQ9KukWSYdWLRv2nKZa1vskrUq1rm8pa0K7IsW8vPoLk6RXSbo17e8qSXtVLdtP0k1pvfPIvsRU7+sISTendX8jaX5tPEMdm07jJNKBJO0M/APwe2BPYC7w4+oyEdEP/ITsg4aaZT8HPgWcl5ok9qla/BaypomtgHtqdw38J7ADsBewM3D6IGGeBZwVEVsDuwLn14njdrJaE8A2EfGS9IF5GfAl4JlkTV2XafO+kqFibIikKcBFwDlkH+LnAq+pKbY92Tf8HYG3A/8laWZELAF+QKpRRMSRwIHAVOCSEcQwEXgr8D/AhprF303LAI5J261ORAuB6yPi3kZ3xyDnTtKewLuBv42IrciS191pvUbO47FsfjyW17zPHcnO6SfIjvXJwE8kzaoq1sg5fR3Z3/MewJFkX6Q+AMwi+6z7l7S/PcjO53vTssuBnypr/psCXAx8L8Xy47TdgVj3A74NvJPs7+8bwKWSptaJZ9hj0wmcRMaPi9M3n0clPQp8dbAywK+Aq8kSwXZp2fo65ddXLW/UORFxa/pmu6l6QUTcERHLIqInIh4m+4D/+/qbYROwm6TtIuKJiPhdg/t/JbAuIr6XYjgX+CPZh8awMY7AC4FJwJciYlNEXAhcX+c9fCwtv5ysZjdYn8p2wCMR0Tswo+qb95OSXlRV9uR0Hp8ga4L8UFXfwoCLgENTDeetZEml2jOpf87rGubc9ZElwOdKmhwRd0fEnVXHoJnzWO3NwOURcXlE9EfEMmAl2RehAY2c0y9HxIZUE78WuC7V9J4iO177pXJHA5el97sJOBOYTlabeyEwGfhiOq8XADdU7eN44BsRcV1E9KX+wJ60Xq08js2Y5yQyfrw6IrYZ+AFOHKLMLhFxYkQ8SdaZCTCnTvk5VcsbNeg329R88KPU7PEY8H0GT1JvJ/vG+MfUJHVEg/vfgb/+JnoPWW1g2BhHYAfg/po2/NrtbqxOCmRNLVtS30ZgO1X1Y0TEQelcbmTz/8Uz0/wtyPq2PivpFdUbS+f2MrJ+r2dGxK/r7K/eOa9rqHMXEXeQfWs/HXgoldshrdrseay2C/CGmi9JB9fE38g5ra6tPVlneuDcbPY3lGrl95L9DdU779V/b7sAJ9XEunNar1Yex2bMcxLpfGuB+4A3VM9U1jn5OmDFIOsN1gE6VMfop9Lyv0lV+DeTNZP89UYi1kXEG4FnAZ8GLpA0Y4htD3iA7B+52lzg/gZjbNR6YEdJ1fHvPIL1a2P4Ldk31kUNbyCzBvg1WQ2s1neBk8g+8GstB16gIa7AqzHkuUv9aQeTHfsgO2etnMdq9wLfq/6SFBEzIuKMqjJ5Pm58s7+hdI53Jvsbqnfe59bE+smaWLdINeLN5HRsxjwnkQ6XvlGdDHxQ0j9KmiZpe+Bssqt/vjDIqhuAeRrZlTBbkTXB/Dm1c79vsIKS3ixpVvoW+Gia3d/APi4H9kjvZZKko4HnAj8bQZwAk9OxGPiZVLP8t2TNOO9O+1kEvGAE299AVQdwRDwKfBT4qqTXS9pK0gRJ+wKDfrBIeg7Zt/J6V6ddTdYH8OXaBanfYRlwkaTnp/ewlaQTJL2tzrYGPXeS9pT0ktTu/xTZt/r+tKzZ81jt+8CRkl4maWI6H4eOIAGO1PnAKyUdpuwS8pPIEvxvyM57L/AvkiZLei2bn/dvAidIOkCZGZJeKWmr2p3kdGzGPCeRLhAR55F1TP4bWTPHH8jagP8uBr/vYqAjfqOkmxrc1UfJLi/+M1lTy4VDlH05cKukJ8g6II9JTTRDSvEeQfaPvxH4D+CIiBhps9zlZB+GAz+n1+ynAryWrEniUbJv5j9j887roXyLrA/hUUkXp21+Bvj3FPOG9PMN4BSyD7ABA1cxlYBfAN9J5TaTaiorIuJPg8Tw+vQ+zyM7J2vImseW1yk71LmbCpxB1vT5INk36/enZU2dx5r3cS9ZDe0DwMNk3/bfR0GfTxGxlux8fpnsPR1Jdnl8peq8Hwv8iaz/5MKqdVcC7wC+AvwfcEcqW0/Lx2Y8UHhQKrOGSLoO+HpEfKfdsZiNFa6JmA1C0t9L2j41BS0G5gM/b3dcZmNJbTuwmf3FnmTt5zOAu4DXR0TDl82adQM3Z5mZWdPcnGVmZk3rquas7bbbLubNm9fuMMzMxpUbb7zxkYiYVW9ZVyWRefPmsXLlynaHYWY2rkga9Bl0bs4yM7OmOYmYmVnTnETMzKxpTiJmZtY0JxEzM2uak4iZmTXNScTMzJrWVfeJmJnV8+s7HuG6uwYbFaEzvOew3Zk8Mf96g5OImXW9T1x2G7etfwzVHYezM5z44t2YPDH/7TqJmFnXe6JnE6/db0c+f/S+7Q5l3HGfiJl1vXJPH9OnFPA1vQs4iZhZ1ytVepkx1Q0zzXASMbOu1tcfPLWpny1cE2lKW5KIpG0lLZO0Lv2eWafMLpJuknSzpFslnVC1bIqkJZJul/RHSa8b3XdgZp2iXOkFYMYU10Sa0a6ayKnAiojYHViRpmutBw6MiH2BA4BTJe2Qlp0GPBQRewDPBa4uPmQz60TlSh8AW0x1TaQZ7Uq9i4BD0+ulwFXAKdUFIqJSNTmVzRPe24DnpHL9wCMFxWlmHa7U45pIK9pVE5kdEevT6weB2fUKSdpZ0irgXuDTEfGApG3S4o+n5q4fS6q7ftrG8ZJWSlr58MMP5/kezKwDPF0TcZ9IUwpLIpKWS1pT52dRdbmICCDqbSMi7o2I+cBuwOKULCYBOwG/iYj9gd8CZw4WR0QsiYgFEbFg1qy6ozuaWRd7uibiq7OaUthRi4iFgy2TtEHSnIhYL2kO8NAw23pA0hrgEOAnQBm4MC3+MfD2nMI2sy5T3uSaSCva1Zx1KbA4vV4MXFJbQNJOkqan1zOBg4G1qebyU/7Sp3IY8IeiAzazzlTuyZKIayLNaVcSOQM4XNI6YGGaRtICSWenMnsB10m6hezqqzMjYnVadgpweuoveQtw0qhGb2Ydo5Qu8XVNpDltSb0RsZGsBlE7fyVwXHq9DJg/yPr3AC8qMkYz6w5lX53VEt+xbmZdreT7RFriJGJmXa1c6WXSBDGlgLE2uoGPmpl1tVJPH1tMmYg6eTCRAjmJmFlXK/sJvi1xEjGzrlaq9PnKrBY4iZhZVyv3uCbSCicRM+tqrom0xknEzLpaudLLFr5HpGlOImbW1co9rom0wknEzLpaqdLru9Vb4CRiZl2t3NPnu9Vb4CRiZl0rIlwTaZGTiJl1rZ7efvrDz81qhZOImXUtj6/eOicRM+taHl+9dU4iZta1BpKI71hvnpOImXUtj2rYOicRM+taHl+9dU4iZta1XBNpnZOImXWtcsVXZ7XKScTMulapx+Ort8pJxMy6lmsirXMSMbOuNVATmT7ZNZFmOYmYWdfKxhKZyIQJanco45aTiJl1rWxUQzdltcJJxMy6Vja+upuyWuEkYmZdq1Tpc39Ii5xEzKxrlSu9vlu9RU4iZta1Sh5fvWVOImbWtcoe1bBlTiJm1rVKHl+9ZU4iZta1XBNpnZOImXWtUsU1kVY5iZhZV+rt66fS2++aSIucRMysK5U3eXz1PDiJmFlX8qiG+XASMbOu5FEN8+EkYmZd6emaiPtEWtKWJCJpW0nLJK1Lv2fWKbOLpJsk3SzpVkknpPlbpXkDP49I+uKovwkzG9eeron46qyWtKsmciqwIiJ2B1ak6VrrgQMjYl/gAOBUSTtExOMRse/AD3APcOEoxW1mHcKjGuajXUlkEbA0vV4KvLq2QERUIqInTU6lTqyS9gCeBVxbTJhm1qlKT3esuybSinYlkdkRsT69fhCYXa+QpJ0lrQLuBT4dEQ/UFDkGOC8iorhQzawTlZ/uWHdNpBWFHT1Jy4Ht6yw6rXoiIkJS3SQQEfcC8yXtAFws6YKI2FBV5BjgLcPEcTxwPMDcuXNH8A7MrJOV3LGei2FrIpL2kLRC0po0PV/SB4dbLyIWRsTedX4uATZImpO2Nwd4aJhtPQCsAQ6pimsfYFJE3DjMuksiYkFELJg1a9ZwYZtZlxioiUz3Jb4taaQ565vA+4FNABGxiqwG0IpLgcXp9WLgktoCknaSND29ngkcDKytKvJG4NwW4zCzLlWq9DF5opgyyXc6tKKRo7dFRFxfM6+3xf2eARwuaR2wME0jaYGks1OZvYDrJN0CXA2cGRGrq7ZxFE4iZtakck+v+0Ny0MgRfETSrkAASHo92eW3TYuIjcBhdeavBI5Lr5cB84fYxrNbicHMulup0scMN2W1rJEk8s/AEuA5ku4H/gd4c6FRmZkVrFzpZQs/N6tlwx7BiLgLWChpBjAhIh4vPiwzs2KVelwTyUMjV2d9StI2EVGKiMclzZT0idEIzsysKOWK+0Ty0EjH+isi4tGBiYj4P+AfCovIzGwUlHr6fLd6DhpJIhMlTR2YSJfdTh2ivJnZmOeaSD4aOYI/AFZI+k6a/if+8twrM7NxqVxxTSQPjXSsfzo9v2rgktyPR8R/FxuWmVmxypU+10Ry0NARjIgrgCsKjsXMbFREBKVKr6/OykEjV2e9Ng0e9WdJj0l6XNJjoxGcmVkRntrUTwS+TyQHjRzBzwBHRsRtRQdjZjYaSk8PSOWaSKsauTprgxOImXWSgfHV3SfSukaO4EpJ5wEXAwMjDRIRHpLWzMalp2sivjqrZY0kka2BMvDSqnmBxzU3s3HKoxrmp5FLfP9pNAIxMxstHl89P8MmEUnTgLcDzwOmDcyPiLcVGJeZWWFcE8lPIx3r3yMbK/1lZIND7QT4Sb5mNm55fPX8NJJEdouIDwGliFgKvBI4oNiwzMyK4/HV89NIEtmUfj8qaW/gGcCzigvJzKxYpYr7RPLSSF1uiaSZwIeAS4EtgQ8XGpWZWYHKPb1IMG2Sk0irGrk66+z08mrA45qb2bhXqvSxxeSJTJigdocy7jVyddY2wFuBedXlI+JfCovKzKxAHl89P40cxcuB3wGrgf5iwzEzK57HV89PI0lkWkT8e+GRmJmNEo9qmJ+G7hOR9A5JcyRtO/BTeGRmZgXx+Or5aSQVV4DPAqeRPTOL9Nud7GY2LpU39bHN9MntDqMjNJJETiK74fCRooMxMxsN5Z5edtxm2vAFbViNNGfdQfYUXzOzjuDx1fPTyFEsATdL+iWbjyfiS3zNbFzy+Or5aSSJXJx+zMw6Qrmnz/eJ5GTIoyhpInBsRLx4lOIxMytUpbefSl+/ayI5GbJPJCL6gH5JzxileMzMCvVkxeOr56mRo/gEsFrSMrL+EcB9ImY2Pnl89Xw1kkQuxOOpm1mH8KiG+WrkKb5LJU0B9kiz1kbEpqHWMTMbqzy+er4aeYrvocBS4G5AwM6SFkfENYVGZmZWgJJrIrlq5Ch+DnhpRKwFkLQHcC7w/CIDMzMrQrlnoGPdNZE8NHLH+uSBBAIQEbcDfuiMmY1Lronkq5GjuFLS2cD30/SbgJXFhWRmVpyyx1fPVSNJ5F3APwMDl/ReC3y1sIjMzApU6nFNJE+DNmdJWpFefiwiPh8Rr00/X4iInsHWa0Qak2SZpHXp98w6ZXaRdJOkmyXdKumEqmVvlLRa0ipJP5e0XSvxmFn3KFfcJ5KnofpE5kg6CHiVpP0k7V/90+J+TwVWRMTuwIo0XWs9cGBE7AscAJwqaQdJk4CzgBdHxHxgFfDuFuMxsy5RqvQyZdIEJk9spEvYhjNUfe7DwIeAnYDP1ywL4CUt7HcRcGh6vRS4Cjhlsx1EVKomp/KXhKf0M0PSRmBrssfVm5kNq+zx1XM1aBKJiAsk/QT4UER8LOf9zo6I9en1g8DseoUk7QxcBuwGvC8iHkjz3wWsJnsMyzqyPpu6JB0PHA8wd+7cvOI3s3Gq5PHVczXcAxgDeEMzG5a0XNKaOj+L6uwj6m0jIu5NTVa7AYslzZY0mayzfz9gB7LmrPcP8R6WRMSCiFgwa9asZt6KmXWQJyseXz1PjaTjmyT9bUTcMJINR8TCwZZJ2iBpTkSslzQHeGiYbT0gaQ1wCHBPmndn2tb51O9TMTP7KyWPapirRnqWDgB+K+nOdDXUakmrWtzvpcDi9HoxcEltAUk7SZqeXs8EDgbWAvcDz5U0UK04HLitxXjMrEuUe3pdE8lRI+n4ZQXs9wzgfElvJ6tZHAUgaQFwQkQcB+wFfE5SkHWknxkRq1O5jwLXSNqU1j+2gBjNrAOVKn3MnDGl3WF0jEae4nuPpIOB3SPiO6kGsGUrO42IjcBhdeavBI5Lr5cB8wdZ/+vA11uJwcy6U9njq+dq2OYsSR8hu/x2oPN6Mn95BIqZ2bhS8vjquWqkT+Q1wKtIoxqmy2y3KjIoM7OiuCaSr0aSSKX6MlxJM4oNycysGP39QdlXZ+WqkSRyvqRvANtIegewHPhmsWGZmeXvyU1+gm/eGulYP1PS4cBjwJ7Ah1Ont5nZuOKxRPI36JGUtDtwJrAr2SNGTo6I+0crMDOzvHlUw/wN1Zz1beBnwOuAG4Evj0pEZmYFcU0kf0Mdya0iYqDvY62km0YjIDOzonhUw/wNlUSmSdqP7G5xgOnV0xHhpGJm44pHNczfUEdyPZuPI/Jg1XSr44mYmY0610TyN9R4Ii8ezUDMzIo2UBOZ4ZpIbjw+pJl1DY+vnj8nETPrGgNXZ83ws7Ny4yRiZl2j3NPHBMHUSf7oy0sjT/Fd0cg8M7OxrlzpY8aUSUgavrA1ZKg71qcBWwDbpZEFB4761sCOoxCbmVmuypVetvCVWbkaqmHwncB7gR3I7lgfSCKPAV8pNiwzs/yVUk3E8jPUJb5nAWdJek9E+JEnZjbulXtcE8lbI71LD0raCkDSByVdKGn/guMyM8tdqdLru9Vz1kgS+VBEPJ7GWV8IfAv4WrFhmZnlL+tYd00kT40kkb70+5XAkoi4DJhSXEhmZsUo9fR6fPWcNZJE7k8jGx4NXC5paoPrmZmNKa6J5K+RZHAU8N/AyyLiUWBb4H1FBmVmVoRSj/tE8jZsEomIMvAQcHCa1QusKzIoM7O8RURWE/HVWblq5I71jwCnAO9PsyYD3y8yKDOzvFX6+untD9dEctZIc9ZrgFcBJYCIeADYqsigzMzy5vHVi9FIEqlERJANRIWkGcWGZGaWv6ef4OuaSK4aSSLnp6uztpH0DmA58M1h1jEzG1OeHkvEfSK5GjYlR8SZkg4ne2bWnsCHI2JZ4ZGZmeXIoxoWo6GjmZLGMknbARuLDcnMLH8e1bAYgzZnSXqhpKvSs7L2k7QGWANskPTy0QvRzKx1T9dEfMd6roY6ml8BPgA8A7gSeEVE/E7Sc4BzgZ+PQnxmZrlwTaQYQ3WsT4qIX0TEj4EHI+J3ABHxx9EJzcwsPwNJxDWRfA2VRPqrXj9ZsywKiMXMrDDldImvayL5Giol7yPpMbIRDaen16TpaYVHZmaWo9LTNxu6JpKnoUY2dLo2s45RrvQybfIEJk7Q8IWtYX6ku5l1hVKl1/eIFMBJxMy6Qrmnz3erF6AtSUTStpKWSVqXfs+sU2YXSTdJulnSrZJOqFp2tKRVaf6nRzd6MxuPXBMpRrtqIqcCKyJid2BFmq61HjgwIvYFDgBOlbSDpGcCnwUOi4jnAdtLOmyU4jazcapc6fOVWQVoVxJZBCxNr5cCr64tEBGViOhJk9VD8j4bWBcRD6fp5cDrigvVzDpBqafX94gUoF1JZHZErE+vHwRm1yskaWdJq4B7gU+nsUzuAPaUNE/SJLIEtPNgO5J0vKSVklY+/PDDgxUzsw7nmkgxCkvLkpYD29dZdFr1RESEpLo3L0bEvcB8STsAF0u6ICI2SHoXcB7ZDZG/AXYdLI6IWAIsAViwYIFvkjTrUu4TKUZhRzQiFg62TNIGSXMiYr2kOWRjuA+1rQfSAyAPAS6IiJ8CP03bOh7oyzF0M+tA5Z4+prsmkrt2NWddCixOrxcDl9QWkLSTpOnp9UzgYGBtmn5W1fwTgbNHIWYzG8dKFfeJFKFdSeQM4HBJ64CFaRpJCyQNJIS9gOsk3QJcDZwZEavTsrMk/QH4NXBGRNw+uuGb2XjS1x88tanffSIFaEtajoiNwF9dlhsRK4Hj0utlwPxB1n9joQGaWUcpe3z1wviOdTPreB5fvThOImbW8Ty+enGcRMys43lUw+I4iZhZx/P46sVxEjGzjlfe5JpIUZxEzKzjlXs8vnpRnETMrOOVPL56YZxEzKzjlX11VmGcRMys45V8n0hhnETMrOOVK71MmiCmTPRHXt58RM2s45V6srFEJLU7lI7jJGJmHa/sJ/gWxknEzDpeyaMaFsZJxMw6XtnjqxfGScTMOp5rIsVxEjGzjleu9LKF7xEphJOImXW8co9rIkVxEjGzjleq9Ppu9YI4iZhZxyv39Plu9YI4iZhZR4sI10QK5CRiZh2tp7ef/vBzs4riJGJmHc3jqxfLScTMOprHVy+Wk4iZdbSBJOI71ovhJGJmHc2jGhbLScTMOprHVy+Wk4iZdTTXRIrlJGJmHa1c8dVZRXISMbOOVurx+OpFchIxs47mmkixnETMrKMN1ESmT3ZNpAhOImbW0bKxRCYyYYLaHUpHchIxs46WjWropqyiOImYWUcr9/T68t4COYmYWUfz+OrFchIxs45WrvT6bvUCOYmYWUcreXz1QjmJmFlHK3tUw0I5iZhZRyt5fPVCtSWJSNpW0jJJ69LvmUOU3VrSfZK+UjXv+ZJWS7pD0pck+QJwM6vLNZFitasmciqwIiJ2B1ak6cF8HLimZt7XgHcAu6eflxcRpJmNf6WKayJFald6XgQcml4vBa4CTqktJOn5wGzg58CCNG8OsHVE/C5Nfxd4NXBFUcEet/QG7tlYLmrzZlagSm8/W0x2TaQo7TqysyNifXr9IFmi2IykCcDngDcDC6sW7QjcVzV9X5pXl6TjgeMB5s6d21Swc7edwZRJ7j4yG4+eM2drXvE327c7jI5VWBKRtByod+ZOq56IiJAUdcqdCFweEfe10uUREUuAJQALFiyot59hffjI5za9fzOzTlZYEomIhYMtk7RB0pyIWJ+apx6qU+xA4BBJJwJbAlMkPQGcBexUVW4n4P4cQzczswa1q43mUmBxer0YuKS2QES8KSLmRsQ84GTguxFxamoGe0zSC9NVWW+tt76ZmRWvXUnkDOBwSevI+jvOAJC0QNLZDax/InA2cAdwJwV2qpuZ2eAU0VQ3wbi0YMGCWLlyZbvDMDMbVyTdGBEL6i3zJUdmZtY0JxEzM2uak4iZmTXNScTMzJrWVR3rkh4G7qmzaDvgkVEOJw+Oe3Q57tHluEffYLHvEhGz6q3QVUlkMJJWDnblwVjmuEeX4x5djnv0NRO7m7PMzKxpTiJmZtY0J5HMknYH0CTHPboc9+hy3KNvxLG7T8TMzJrmmoiZmTXNScTMzJrW1UlE0sslrZV0h6ShxnkfUyTdLWm1pJsljeknSkr6tqSHJK2pmretpGWS1qXfM9sZYz2DxH26pPvTcb9Z0j+0M8Z6JO0s6ZeS/iDpVkn/muaP6WM+RNxj+phLmibpekm3pLg/mub/P0nXpc+W8yRNaXes1YaI+xxJ/1N1vPcddlvd2iciaSJwO3A42RC7NwBvjIg/tDWwBki6G1gQEWP+hiZJLwKeIBsPZu807zPAnyLijJS8Z0bEKe2Ms9YgcZ8OPBERZ7YztqGkQd7mRMRNkrYCbgReDRzLGD7mQ8R9FGP4mKcxjWZExBOSJgO/Av4V+Hfgwoj4kaSvA7dExNfaGWu1IeI+AfhZRFzQ6La6uSbyAuCOiLgrIirAj4BFbY6p40TENcCfamYvApam10vJPizGlEHiHvMiYn1E3JRePw7cBuzIGD/mQ8Q9pkXmiTQ5Of0E8BJg4IN4LB7vweIesW5OIjsC91ZN38c4+KNNAviFpBslHd/uYJowO41QCfAgMLudwYzQuyWtSs1dY6pJqJakecB+wHWMo2NeEzeM8WMuaaKkm8mG+V5GNlDeoxHRm4qMyc+W2rgjYuB4fzId7y9Imjrcdro5iYxnB0fE/sArgH9OTS/jUmTtqeOlTfVrwK7AvsB64HNtjWYIkrYEfgK8NyIeq142lo95nbjH/DGPiL6I2BfYiayF4zntjagxtXFL2ht4P1n8fwtsCwzb5NnNSeR+YOeq6Z3SvDEvIu5Pvx8CLiL7wx1PNqQ28IG28IfaHE9DImJD+sfrB77JGD3uqY37J8APIuLCNHvMH/N6cY+XYw4QEY8CvwQOBLaRNCktGtOfLVVxvzw1K0ZE9ADfoYHj3c1J5AZg93QVxRTgGODSNsc0LEkzUscjkmYALwXWDL3WmHMpsDi9Xgxc0sZYGjbwIZy8hjF43FOH6beA2yLi81WLxvQxHyzusX7MJc2StE16PZ3sQp3byD6UX5+KjcXjXS/uP1Z90RBZP86wx7trr84CSJcLfhGYCHw7Ij7Z3oiGJ+nZZLUPgEnAD8dy3JLOBQ4le8T0BuAjwMXA+cBcskfzHxURY6oTe5C4DyVrVgngbuCdVf0MY4Kkg4FrgdVAf5r9AbL+hTF7zIeI+42M4WMuaT5Zx/lEsi/l50fEx9L/6Y/ImoR+D7w5fbsfE4aI+0pgFiDgZuCEqg74+tvq5iRiZmat6ebmLDMza5GTiJmZNc1JxMzMmuYkYmZmTXMSMTOzpjmJWFeQtL2kH0m6Mz0u5nJJexS4v0Ml/WyYMvs281RaSVdJWlAz76L01NU7JP256imsB1WV2UHSkA/WkzRPVU8uNhvOpOGLmI1v6capi4ClEXFMmrcP2fOjbm9jaPsCC4DLW91QRLwGsuQFnBwRR1QvlzQpIh7gLzfAmeXCNRHrBi8GNkXE1wdmRMQtEXGtMp+VtEbZGC1Hw9M1iaslXSLpLklnSHqTsjEYVkvaNZU7R9LXJa2UdLukI2p3np4y8O207u8lLUpPSfgYcHSqMRxdr1xaf3qqRd0m6SJgeiNvWtKxki5NN5CtqK5lpNfXSrop/RxUZ/3npVhuTg/k233ER946nmsi1g32Jhufop7XktUI9iG7O/0GSdekZfsAe5E9Ev4u4OyIeIGyAZPeA7w3lZtH9oyhXYFfStqtZh+nAVdGxNvSoyauB5YDHyYbF+bdAJI+VVtO0nLgnUA5IvZKdxrfNIL3vj8wPyL+pOzpuAMeAg6PiKdScjiXrFZU7QTgrIj4QUp6E0ewX+sSTiLW7Q4Gzo2IPrKHFF5N9gTTx4AbBh6xIelO4BdpndVktZsB56cHBK6TdBd//RTXlwKvknRymp5G9viRWoOVexHwJYCIWCVp1Qje37JBHm8yGfiKspHr+oB6/UO/BU6TtBPZAEvrRrBf6xJOItYNbqW5voDqZx31V033s/n/Tu2zg2qnBbwuItZuNlM6oMFyI4m5VmmQ+f9G9kywfciatZ+qLRARP5R0HfBK4HJJ74yIK1sJxjqP+0SsG1wJTFXVAF6S5ks6hOyhf0crG6BnFtm3/utHuP03SJqQ+kmeDaytWf7fwHtSBz+S9kvzHwe2aqDcNcA/pnl7A/NHGF89zwDWpxrUW6jTVJUeInhXRHyJ7Cm0eezXOoyTiHW8NAjTa4CF6RLfW4H/JBvh7yJgFXALWbL5j4h4cIS7+F+yxHMF2VNPa7/Vf5ys+WhV2vfH0/xfAs8d6FgfotzXgC0l3UbWGT9Y/85IfBVYLOkWsua3ejWWo4A1yka/2xv4bg77tQ7jp/iatUDSOcDPImLI+y/MOpVrImZm1jTXRMzMrGmuiZiZWdOcRMzMrGlOImZm1jQnETMza5qTiJmZNe3/A3nk5gjfM1vNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Details of Hyperparameter optimization ***\n",
      "HPO for LightGBMClassifier model:  Num. configurations tried = 34, Time spent = 91.04652786254883, Search strategy = random\n",
      "Best hyperparameter-configuration (validation-performance: accuracy = -0.35833333333333334):\n",
      "{'boosting_type▁choice': 0, 'feature_fraction': 0.8697498361349312, 'learning_rate': 0.06142885606149031, 'min_data_in_leaf': 22, 'num_leaves': 47}\n",
      "*** End of fit() summary ***\n"
     ]
    }
   ],
   "source": [
    "results = predictor_hpo.fit_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "Erickson et al. [**AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data**](https://arxiv.org/abs/2003.06505). *Arxiv*, 2020.\n",
    "\n",
    "Boehmke B, Greenwell B. [**Hands-On Machine Learning with R** (Chapters 8-13 and 15)](https://bradleyboehmke.github.io/HOML/). 2020.\n",
    "\n",
    "Dietterich T. [**Ensemble Methods in Machine Learning**](https://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf). In: *Multiple Classifier Systems*, 2000. \n",
    "\n",
    "Rocca J. [**Ensemble methods: bagging, boosting and stacking**](https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205). *Towards Data Science*, 2019.\n",
    "\n",
    "Caruana et al. [**Ensemble Selection from Libraries of Models**](https://www.cs.cornell.edu/~alexn/papers/shotgun.icml04.revised.rev2.pdf). In: *ICML*, 2004.\n",
    "\n",
    "van Veen et al. [**Kaggle Ensembling Guide**](https://mlwave.com/kaggle-ensembling-guide/). *mlwave.com*, 2015.\n",
    "\n",
    "Breiman L. [**Bagging Predictors**](https://www.stat.berkeley.edu/~breiman/bagging.pdf). *Technical Report*, 1994.\n",
    "\n",
    "Breiman L. [**Stacked Regressions**](https://statistics.berkeley.edu/sites/default/files/tech-reports/367.pdf). *Technical Report*, 1996.\n",
    "\n",
    " Wolpert, D. [**Stacked generalization**](https://www.sciencedirect.com/science/article/abs/pii/S0893608005800231). *Neural Networks*, 1992.\n",
    "\n",
    "Van der Laan et al. [**Super Learner**](https://biostats.bepress.com/cgi/viewcontent.cgi?article=1226&context=ucbbiostat). *Stat Appl Genet Mol Biol*, 2007.\n",
    "\n",
    "Polley E, Van der Laan, M. [**Super Learner In Prediction**](https://biostats.bepress.com/cgi/viewcontent.cgi?article=1269&context=ucbbiostat). *U.C. Berkeley Division of Biostatistics Working Paper Series*, 2010.\n",
    "\n",
    "Koren Y. [**The BellKor Solution to the Netflix Grand Prize**](https://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf). *netflixprize.com*, 2009.\n",
    "\n",
    "Titericz G, Semenov S. [**1st PLACE - WINNER SOLUTION**](https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335). *Kaggle Otto Group Product Classification Challenge*, 2016.\n",
    "\n",
    "Ke et al. [**LightGBM: A Highly Efficient Gradient Boosting Decision Tree**](https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf). In: *NIPS*, 2017.\n",
    "\n",
    "Prokhorenkova et al. [**CatBoost: unbiased boosting with categorical features**](https://arxiv.org/abs/1706.09516). In: *NeurIPS*, 2018."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
